[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:55 dist-test-slave-s1kg krb5kdc[28574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:55 dist-test-slave-s1kg krb5kdc[28574](info): set up 2 sockets
Sep 28 02:51:55 dist-test-slave-s1kg krb5kdc[28574](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 02:51:56.787918 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.012s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:51:56 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:51:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:51:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:51:56.971055 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:43589
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:57.299185 28587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:57.299492 28587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:57.299683 28587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:57.307207 28587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:51:57.307328 28587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:57.307421 28587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:57.307510 28587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:51:57.307591 28587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:51:57.317678 28587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:43589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:51:57.319135 28587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:57.320506 28587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:57.333137 28593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:57.333237 28587 server_base.cc:1034] running on GCE node
W20260928 02:51:57.333217 28595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:57.333571 28592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:57.334471 28587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:57.335958 28587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:57.337176 28587 hybrid_clock.cc:648] HybridClock initialized: now 1790563917337119 us; error 35 us; skew 500 ppm
I20260928 02:51:57.337591 28587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:51:57 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563917, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:57.340826 28587 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:51:57.342489 28587 webserver.cc:460] Webserver started at http://127.27.229.254:40487/ using document root <none> and password file <none>
I20260928 02:51:57.343220 28587 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:57.343323 28587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:57.343600 28587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:57.346681 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "54d3f5a7257f46d7bc23ac498a666def"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "045b2d5415be04258bbd46666279375c"
server_key_iv: "8f96db83c6c6cc6c8206b11e6155d6f4"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.347628 28587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "54d3f5a7257f46d7bc23ac498a666def"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "045b2d5415be04258bbd46666279375c"
server_key_iv: "8f96db83c6c6cc6c8206b11e6155d6f4"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.353330 28587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:51:57.356878 28602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:57.358281 28587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:57.358500 28587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "54d3f5a7257f46d7bc23ac498a666def"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "045b2d5415be04258bbd46666279375c"
server_key_iv: "8f96db83c6c6cc6c8206b11e6155d6f4"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.358763 28587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:57.383335 28587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:57.388103 28587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:57.388350 28587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:57.408222 28587 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:43589
I20260928 02:51:57.408257 28653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:43589 every 8 connection(s)
I20260928 02:51:57.409691 28587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:51:57.413622 28654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:51:57.416378 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 28587
I20260928 02:51:57.416700 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:51:57.418555 28567 external_mini_cluster.cc:1428] Setting key 2e71077e3f942e0fa1976c4c48531d76
I20260928 02:51:57.424736 28654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def: Bootstrap starting.
I20260928 02:51:57.427793 28654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def: Neither blocks nor log segments found. Creating new log.
I20260928 02:51:57.428790 28654 log.cc:824] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def: Log is configured to *not* fsync() on all Append() calls
I20260928 02:51:57.431574 28654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def: No bootstrap required, opened a new log
I20260928 02:51:57.440155 28654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } }
I20260928 02:51:57.440609 28654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:51:57.440733 28654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54d3f5a7257f46d7bc23ac498a666def, State: Initialized, Role: FOLLOWER
I20260928 02:51:57.441480 28654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } }
I20260928 02:51:57.441740 28654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:51:57.441900 28654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:51:57.442096 28654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:51:57.444653 28654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } }
I20260928 02:51:57.445165 28654 leader_election.cc:302] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54d3f5a7257f46d7bc23ac498a666def; no voters:
I20260928 02:51:57.445822 28654 leader_election.cc:288] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:51:57.445941 28659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:51:57.447113 28659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [term 1 LEADER]: Becoming Leader. State: Replica: 54d3f5a7257f46d7bc23ac498a666def, State: Running, Role: LEADER
Sep 28 02:51:57 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:51:57.447700 28659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } }
I20260928 02:51:57.448312 28654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:51:57.451252 28660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54d3f5a7257f46d7bc23ac498a666def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } } }
I20260928 02:51:57.451705 28660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:57.463788 28664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:51:57.463994 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:57.422449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41850 (local address 127.27.229.254:43589)
0928 02:51:57.423373 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:57.423388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:57.424815 (+ 1427us) server_negotiation.cc:407] Connection header received
0928 02:51:57.433592 (+ 8777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:57.433612 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:57.434000 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:57.434414 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:57.437346 (+ 2932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:57.439062 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:57.441120 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:57.442005 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:57.442221 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:57.449087 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:57.449096 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:57.449114 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:57.449171 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:57.454073 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:57.455733 (+ 1660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:57.455747 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:57.455770 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:57.456048 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:57.456889 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:57.456897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:57.456903 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:57.457837 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:57.461314 (+ 3477us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:57.461709 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 02:51:57.462312 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":412,"threads_started":1}
I20260928 02:51:57.466073 28661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54d3f5a7257f46d7bc23ac498a666def. Latest consensus state: current_term: 1 leader_uuid: "54d3f5a7257f46d7bc23ac498a666def" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54d3f5a7257f46d7bc23ac498a666def" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43589 } } }
I20260928 02:51:57.466471 28661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def [sys.catalog]: This master's current role is: LEADER
I20260928 02:51:57.470993 28664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:51:57.479473 28664 catalog_manager.cc:1347] Generated new cluster ID: 9f526e6813104a28bdb0466f2445aab8
I20260928 02:51:57.479591 28664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:51:57.496554 28664 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:51:57.498059 28664 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:51:57.528365 28664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54d3f5a7257f46d7bc23ac498a666def: Generated new TSK 0
I20260928 02:51:57.529469 28664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:51:57.586548 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:57.932811 28680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:57.933143 28680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:57.933312 28680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:57.940825 28680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:57.940994 28680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:57.941171 28680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:51:57.951833 28680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:51:57.953187 28680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:57.954586 28680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:57.969954 28685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:57.970155 28686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:57.970137 28688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:57.972419 28680 server_base.cc:1034] running on GCE node
I20260928 02:51:57.973176 28680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:57.974197 28680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:57.975420 28680 hybrid_clock.cc:648] HybridClock initialized: now 1790563917975348 us; error 68 us; skew 500 ppm
I20260928 02:51:57.975828 28680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:51:57 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563917, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:57.979207 28680 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:51:57.980724 28680 webserver.cc:460] Webserver started at http://127.27.229.193:33987/ using document root <none> and password file <none>
I20260928 02:51:57.981455 28680 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:57.981611 28680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:57.981935 28680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:57.985087 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "101ce58bcb5c426faa017acfb7c7b4ae"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "8424c7854ee29353a0024cca58b1729c"
server_key_iv: "4cbe2e5817c51a9c9d6e61e431c582ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.985858 28680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "101ce58bcb5c426faa017acfb7c7b4ae"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "8424c7854ee29353a0024cca58b1729c"
server_key_iv: "4cbe2e5817c51a9c9d6e61e431c582ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.991273 28680 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:51:57.994964 28695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:57.996418 28680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:51:57.996675 28680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "101ce58bcb5c426faa017acfb7c7b4ae"
format_stamp: "Formatted at 2026-09-28 02:51:57 on dist-test-slave-s1kg"
server_key: "8424c7854ee29353a0024cca58b1729c"
server_key_iv: "4cbe2e5817c51a9c9d6e61e431c582ba"
server_key_version: "encryptionkey@0"
I20260928 02:51:57.996902 28680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:58.037012 28680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:58.041972 28680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:58.042289 28680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:58.043404 28680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:58.044829 28680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:58.044932 28680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:58.045087 28680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:58.045161 28680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:58.080169 28680 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:40655
I20260928 02:51:58.080225 28807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:40655 every 8 connection(s)
I20260928 02:51:58.081528 28680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:51:58.085633 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 28680
I20260928 02:51:58.085878 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:51:58.086694 28567 external_mini_cluster.cc:1428] Setting key ae0eedaf64c8b9798a2866e0729b58b6
Sep 28 02:51:58 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563917, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:51:58.105062 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:58.084275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39019 (local address 127.27.229.254:43589)
0928 02:51:58.084756 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:58.084762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:58.086054 (+ 1292us) server_negotiation.cc:407] Connection header received
0928 02:51:58.088467 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:58.088473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:58.088550 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:58.088683 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:58.090955 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.091693 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.094165 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.094584 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.094731 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:58.098438 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:58.098444 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:58.098447 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:58.098482 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:58.101591 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:58.102539 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:58.102547 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:58.102551 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:58.102693 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:58.103177 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:58.103184 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:58.103187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:58.103466 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:58.103637 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:58.104440 (+ 803us) server_negotiation.cc:299] Negotiation successful
0928 02:51:58.104686 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":271,"threads_started":1}
I20260928 02:51:58.106163 28811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:58.084896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:43589 (local address 127.27.229.193:39019)
0928 02:51:58.085790 (+ 894us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:58.085851 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:58.088162 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:58.088954 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:58.088963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:58.089761 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:58.090658 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:58.090688 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.091888 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.091897 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:58.093984 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:58.093998 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.094863 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.094871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:58.095126 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:58.095896 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:58.095924 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:58.098191 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:58.101756 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:58.101766 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:58.101811 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:58.102396 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:58.102814 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:58.102819 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:58.102823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:58.103052 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:58.103602 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:58.103610 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:58.104003 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 02:51:58.104344 (+ 341us) client_negotiation.cc:240] Negotiation successful
0928 02:51:58.104940 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:51:58.108050 28808 heartbeater.cc:347] Connected to a master server at 127.27.229.254:43589
I20260928 02:51:58.108399 28808 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:58.109162 28808 heartbeater.cc:511] Master 127.27.229.254:43589 requested a full tablet report, sending...
I20260928 02:51:58.111832 28619 ts_manager.cc:194] Registered new tserver with Master: 101ce58bcb5c426faa017acfb7c7b4ae (127.27.229.193:40655)
I20260928 02:51:58.113418 28619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:39019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:51:58.131248 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:58.461162 28814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:58.461449 28814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:58.461545 28814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:58.468894 28814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:58.469007 28814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:58.469120 28814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:51:58.478901 28814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:51:58.480154 28814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:58.481465 28814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:58.497702 28820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:58.497741 28819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:58.497658 28822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:58.498678 28814 server_base.cc:1034] running on GCE node
I20260928 02:51:58.499395 28814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:58.500420 28814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:58.501682 28814 hybrid_clock.cc:648] HybridClock initialized: now 1790563918501620 us; error 53 us; skew 500 ppm
I20260928 02:51:58.502044 28814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:51:58 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563918, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:58.505663 28814 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:51:58.507259 28814 webserver.cc:460] Webserver started at http://127.27.229.194:40829/ using document root <none> and password file <none>
I20260928 02:51:58.507943 28814 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:58.508050 28814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:58.508304 28814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:58.511318 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "fd869b6ea3794448a1cf8de46f647bb3"
format_stamp: "Formatted at 2026-09-28 02:51:58 on dist-test-slave-s1kg"
server_key: "03574db4402595c95f9a7d1a3557c898"
server_key_iv: "520993a24e644a16b325c50726551881"
server_key_version: "encryptionkey@0"
I20260928 02:51:58.512053 28814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "fd869b6ea3794448a1cf8de46f647bb3"
format_stamp: "Formatted at 2026-09-28 02:51:58 on dist-test-slave-s1kg"
server_key: "03574db4402595c95f9a7d1a3557c898"
server_key_iv: "520993a24e644a16b325c50726551881"
server_key_version: "encryptionkey@0"
I20260928 02:51:58.517455 28814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:51:58.521188 28829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:58.522668 28814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:51:58.522964 28814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "fd869b6ea3794448a1cf8de46f647bb3"
format_stamp: "Formatted at 2026-09-28 02:51:58 on dist-test-slave-s1kg"
server_key: "03574db4402595c95f9a7d1a3557c898"
server_key_iv: "520993a24e644a16b325c50726551881"
server_key_version: "encryptionkey@0"
I20260928 02:51:58.523192 28814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:58.538487 28814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:58.543172 28814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:58.543496 28814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:58.544512 28814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:58.545922 28814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:58.545994 28814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:58.546146 28814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:58.546200 28814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:58.578658 28814 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41065
I20260928 02:51:58.578755 28941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41065 every 8 connection(s)
I20260928 02:51:58.580010 28814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:51:58.589357 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 28814
I20260928 02:51:58.589576 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:51:58.590000 28567 external_mini_cluster.cc:1428] Setting key 297d679e6a0fbfe375b057301f7de2b2
Sep 28 02:51:58 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563918, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:51:58.600311 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:58.582657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37677 (local address 127.27.229.254:43589)
0928 02:51:58.582926 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:58.582931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:58.584183 (+ 1252us) server_negotiation.cc:407] Connection header received
0928 02:51:58.585970 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:58.585974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:58.586052 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:58.586141 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:58.587772 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.588472 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.590219 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.590643 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.590783 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:58.594477 (+ 3694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:58.594482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:58.594485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:58.594518 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:58.597216 (+ 2698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:58.597967 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:58.597979 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:58.597983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:58.598100 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:58.598534 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:58.598539 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:58.598542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:58.598803 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:58.599007 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:58.599701 (+ 694us) server_negotiation.cc:299] Negotiation successful
0928 02:51:58.599924 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:51:58.601539 28944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:58.583218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:43589 (local address 127.27.229.194:37677)
0928 02:51:58.584011 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:58.584049 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:58.585700 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:58.586410 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:58.586418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:58.587014 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:58.587601 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:58.587619 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.588630 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.588635 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:58.590036 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:58.590069 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:58.590896 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:58.590904 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:58.591243 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:58.592065 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:58.592098 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:58.594247 (+ 2149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:58.597385 (+ 3138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:58.597394 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:58.597414 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:58.597838 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:58.598214 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:58.598219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:58.598222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:58.598415 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:58.598984 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:58.598992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:58.599344 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 02:51:58.599614 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 02:51:58.600169 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":289,"threads_started":1}
I20260928 02:51:58.603149 28942 heartbeater.cc:347] Connected to a master server at 127.27.229.254:43589
I20260928 02:51:58.603493 28942 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:58.604211 28942 heartbeater.cc:511] Master 127.27.229.254:43589 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:51:58.605705 28619 ts_manager.cc:194] Registered new tserver with Master: fd869b6ea3794448a1cf8de46f647bb3 (127.27.229.194:41065)
I20260928 02:51:58.606518 28619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:37677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:51:58.630244 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:51:58.968254 28947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:51:58.968546 28947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:51:58.968642 28947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:51:58.975920 28947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:51:58.976075 28947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:51:58.976250 28947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:51:58.985962 28947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:43589
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:51:58.987339 28947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:51:58.988608 28947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:51:59.001667 28953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:59.001721 28952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:51:59.001929 28955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:51:59.003054 28947 server_base.cc:1034] running on GCE node
I20260928 02:51:59.003750 28947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:51:59.005117 28947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:51:59.006372 28947 hybrid_clock.cc:648] HybridClock initialized: now 1790563919006301 us; error 81 us; skew 500 ppm
I20260928 02:51:59.006736 28947 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:51:59.010336 28947 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:51:59.012171 28947 webserver.cc:460] Webserver started at http://127.27.229.195:42163/ using document root <none> and password file <none>
I20260928 02:51:59.012914 28947 fs_manager.cc:359] Metadata directory not provided
I20260928 02:51:59.013064 28947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:51:59.013379 28947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:51:59.016561 28947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "e4c17dc96ea84b4eb1729f57bb67fac8"
format_stamp: "Formatted at 2026-09-28 02:51:59 on dist-test-slave-s1kg"
server_key: "7849696d40e8a45c54af1d4c7413adf4"
server_key_iv: "5efc0865709c2a7cc7df3dbdddfb54f0"
server_key_version: "encryptionkey@0"
I20260928 02:51:59.017381 28947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "e4c17dc96ea84b4eb1729f57bb67fac8"
format_stamp: "Formatted at 2026-09-28 02:51:59 on dist-test-slave-s1kg"
server_key: "7849696d40e8a45c54af1d4c7413adf4"
server_key_iv: "5efc0865709c2a7cc7df3dbdddfb54f0"
server_key_version: "encryptionkey@0"
I20260928 02:51:59.023032 28947 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:51:59.026665 28962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:59.028303 28947 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:51:59.028448 28947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "e4c17dc96ea84b4eb1729f57bb67fac8"
format_stamp: "Formatted at 2026-09-28 02:51:59 on dist-test-slave-s1kg"
server_key: "7849696d40e8a45c54af1d4c7413adf4"
server_key_iv: "5efc0865709c2a7cc7df3dbdddfb54f0"
server_key_version: "encryptionkey@0"
I20260928 02:51:59.028637 28947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:51:59.047641 28947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:51:59.052196 28947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:51:59.052456 28947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:51:59.053421 28947 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:51:59.054733 28947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:51:59.054805 28947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:59.054965 28947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:51:59.055014 28947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:51:59.091259 28947 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43209
I20260928 02:51:59.091305 29074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43209 every 8 connection(s)
I20260928 02:51:59.092547 28947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:51:59.098762 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 28947
I20260928 02:51:59.098978 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:51:59.099511 28567 external_mini_cluster.cc:1428] Setting key 526343476ac28e767e8537665e3987de
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:51:59.113559 28810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.095200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:60141 (local address 127.27.229.254:43589)
0928 02:51:59.095374 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.095378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.096444 (+ 1066us) server_negotiation.cc:407] Connection header received
0928 02:51:59.098207 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.098211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.098280 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.098381 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.100522 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.101561 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.103759 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.104238 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.104338 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.107742 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.107746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.107750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.107782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.110551 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.111379 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.111383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.111386 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.111528 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.111957 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.111961 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.111963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.112225 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.112344 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.113127 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.113292 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:51:59.114575 29077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.095595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:43589 (local address 127.27.229.195:60141)
0928 02:51:59.096270 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 02:51:59.096311 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:51:59.097926 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:51:59.098714 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:51:59.098726 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:51:59.099468 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:51:59.100293 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:59.100318 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.101723 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.101732 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:59.103609 (+ 1877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:51:59.103624 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.104443 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.104448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:51:59.104646 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.105321 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:51:59.105347 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:51:59.107535 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:51:59.110699 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:59.110706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:59.110719 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:59.111223 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:59.111635 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:51:59.111638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:51:59.111640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:51:59.111813 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:51:59.112395 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:51:59.112401 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:51:59.112752 (+ 351us) client_negotiation.cc:769] Sending connection context
0928 02:51:59.113001 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 02:51:59.113400 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":257,"threads_started":1}
I20260928 02:51:59.116240 29075 heartbeater.cc:347] Connected to a master server at 127.27.229.254:43589
I20260928 02:51:59.116580 29075 heartbeater.cc:464] Registering TS with master...
I20260928 02:51:59.116684 28808 heartbeater.cc:503] Master 127.27.229.254:43589 was elected leader, sending a full tablet report...
I20260928 02:51:59.117503 29075 heartbeater.cc:511] Master 127.27.229.254:43589 requested a full tablet report, sending...
I20260928 02:51:59.118988 28619 ts_manager.cc:194] Registered new tserver with Master: e4c17dc96ea84b4eb1729f57bb67fac8 (127.27.229.195:43209)
I20260928 02:51:59.119765 28619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:60141
I20260928 02:51:59.129107 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:51:59.176823 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.160299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.27.229.193:40655)
0928 02:51:59.161133 (+ 834us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.161145 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.161181 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:51:59.161261 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.161268 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.161555 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.161731 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.163293 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.164312 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.165759 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.166363 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.166517 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.170907 (+ 4390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.170921 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.170937 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.170979 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.173955 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.174698 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.174706 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.174709 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.174920 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.175336 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.175340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.175342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.175744 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.175988 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.176310 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.176543 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":463,"threads_started":1}
W20260928 02:51:59.178267 28723 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55928
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:51:59.224293 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.208434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55930 (local address 127.27.229.193:40655)
0928 02:51:59.208709 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.208714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.208869 (+ 155us) server_negotiation.cc:407] Connection header received
0928 02:51:59.209143 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.209148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.209230 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.209320 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.210763 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.211542 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.212864 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.213380 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.213504 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.218308 (+ 4804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.218317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.218322 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.218372 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.221446 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.222143 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.222147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.222150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.222275 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.222797 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.222803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.222810 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.223218 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.223512 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.223852 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.224054 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 02:51:59.226006 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 28680
I20260928 02:51:59.239320 29082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.226991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55946 (local address 127.27.229.193:40655)
0928 02:51:59.227208 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.227217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.227483 (+ 266us) server_negotiation.cc:407] Connection header received
0928 02:51:59.227603 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.227606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.227664 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.227758 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.228927 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.229621 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.231038 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.231492 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.231594 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.234063 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.234067 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.234070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.234102 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.236773 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.237450 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.237454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.237457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.237607 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.237960 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.237964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.237966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.238240 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.238458 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.238712 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.238951 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:51:59.240103 28803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:51:59.324316 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 28680
I20260928 02:51:59.341312 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 28814
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:51:59.358255 29089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.342404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.27.229.194:41065)
0928 02:51:59.342949 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.342956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.342981 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:51:59.343056 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.343062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.343278 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.343405 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.344783 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.345887 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.347223 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.347815 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.347920 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.352478 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.352489 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.352503 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.352562 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.355490 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.356181 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.356188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.356191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.356316 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.356765 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.356769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.356771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.357109 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.357344 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.357640 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.357834 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260928 02:51:59.359918 28937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:51:59.448712 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 28814
I20260928 02:51:59.470562 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 28947
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[28574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:51:59.487712 29091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:51:59.471648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49622 (local address 127.27.229.195:43209)
0928 02:51:59.472290 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 02:51:59.472298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:51:59.472328 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:51:59.472392 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:51:59.472441 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:51:59.472662 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:51:59.472804 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:51:59.474015 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.475143 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.476590 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:51:59.477209 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:51:59.477323 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:51:59.481913 (+ 4590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:51:59.481926 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:51:59.481945 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:51:59.481984 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:51:59.484852 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.485558 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.485566 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.485568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.485703 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:51:59.486025 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:51:59.486029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:51:59.486031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:51:59.486402 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:51:59.486634 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:51:59.486978 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 02:51:59.487220 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":361,"threads_started":1}
I20260928 02:51:59.489069 29070 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:51:59.574481 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 28947
I20260928 02:51:59.591454 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 28587
I20260928 02:51:59.592770 28649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:51:59.695495 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 28587
2026-09-28T02:51:59Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4011 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[29096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[29096](info): set up 2 sockets
Sep 28 02:51:59 dist-test-slave-s1kg krb5kdc[29096](info): commencing operation
krb5kdc: starting...
W20260928 02:52:00.822960 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:00 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:01.014686 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33103
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:33103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:01.337584 29109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:01.337934 29109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:01.338088 29109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:01.345703 29109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:01.345868 29109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:01.345992 29109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:01.346091 29109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:01.346176 29109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:01.356377 29109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:33103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:01.357776 29109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:01.359340 29109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:01.373919 29114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:01.373951 29115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:01.373941 29117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:01.375065 29109 server_base.cc:1034] running on GCE node
I20260928 02:52:01.375906 29109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:01.377770 29109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:01.379027 29109 hybrid_clock.cc:648] HybridClock initialized: now 1790563921378942 us; error 69 us; skew 500 ppm
I20260928 02:52:01.379475 29109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:01 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563921, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:01.386098 29109 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:01.387897 29109 webserver.cc:460] Webserver started at http://127.27.229.254:35541/ using document root <none> and password file <none>
I20260928 02:52:01.388680 29109 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:01.388837 29109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:01.389179 29109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:01.392416 29109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "9c00940bb0654ed4b61d05d92b0ff4c4"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "b2c425e6553c62cbea4d3744b74cd093"
server_key_iv: "dfd15a624b2c27e7351ca69fab36f5fb"
server_key_version: "encryptionkey@0"
I20260928 02:52:01.393226 29109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "9c00940bb0654ed4b61d05d92b0ff4c4"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "b2c425e6553c62cbea4d3744b74cd093"
server_key_iv: "dfd15a624b2c27e7351ca69fab36f5fb"
server_key_version: "encryptionkey@0"
I20260928 02:52:01.398904 29109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:01.402567 29124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:01.404052 29109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:52:01.404241 29109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "9c00940bb0654ed4b61d05d92b0ff4c4"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "b2c425e6553c62cbea4d3744b74cd093"
server_key_iv: "dfd15a624b2c27e7351ca69fab36f5fb"
server_key_version: "encryptionkey@0"
I20260928 02:52:01.404454 29109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:01.434070 29109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:01.439102 29109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:01.439502 29109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:01.459192 29109 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:33103
I20260928 02:52:01.459234 29175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:33103 every 8 connection(s)
I20260928 02:52:01.460628 29109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:01.462951 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29109
I20260928 02:52:01.463191 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:01.464008 28567 external_mini_cluster.cc:1428] Setting key 98ee0fcc7f1648e1c0671d6e9d66fab9
I20260928 02:52:01.465294 29176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:01.477965 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4: Bootstrap starting.
Sep 28 02:52:01 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:01.481639 29176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:01.482903 29176 log.cc:824] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:01.485862 29176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4: No bootstrap required, opened a new log
I20260928 02:52:01.488157 29179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:01.466198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57232 (local address 127.27.229.254:33103)
0928 02:52:01.467083 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:01.467095 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:01.467138 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:52:01.468686 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:01.468709 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:01.469125 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:01.469546 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:01.471129 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:01.472458 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:01.473707 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:01.474734 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:01.475025 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:01.479555 (+ 4530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:01.479564 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:01.479585 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:01.479639 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:01.483369 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:01.484224 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:01.484233 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:01.484241 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:01.484480 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:01.484918 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:01.484930 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:01.484932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:01.485537 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:01.485861 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:01.486479 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 02:52:01.486998 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":360,"threads_started":1}
I20260928 02:52:01.494378 29176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } }
I20260928 02:52:01.494939 29176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:01.495078 29176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c00940bb0654ed4b61d05d92b0ff4c4, State: Initialized, Role: FOLLOWER
I20260928 02:52:01.495746 29176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } }
I20260928 02:52:01.495949 29176 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:01.496038 29176 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:01.496169 29176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:01.498263 29176 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } }
I20260928 02:52:01.498775 29176 leader_election.cc:302] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c00940bb0654ed4b61d05d92b0ff4c4; no voters:
I20260928 02:52:01.499501 29176 leader_election.cc:288] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:01.499653 29181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:01.500864 29181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [term 1 LEADER]: Becoming Leader. State: Replica: 9c00940bb0654ed4b61d05d92b0ff4c4, State: Running, Role: LEADER
I20260928 02:52:01.501369 29181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } }
I20260928 02:52:01.501768 29176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:01.504462 29183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c00940bb0654ed4b61d05d92b0ff4c4. Latest consensus state: current_term: 1 leader_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } } }
I20260928 02:52:01.504436 29182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c00940bb0654ed4b61d05d92b0ff4c4" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33103 } } }
I20260928 02:52:01.505014 29182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:01.504905 29183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:01.506278 29190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:01.515107 29190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:01.523906 29190 catalog_manager.cc:1347] Generated new cluster ID: eba27ba43e034a8fb18ef29feb317910
I20260928 02:52:01.524040 29190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:01.537894 29190 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:01.539304 29190 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:01.554349 29190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c00940bb0654ed4b61d05d92b0ff4c4: Generated new TSK 0
I20260928 02:52:01.555545 29190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:01.608038 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:01.949877 29202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:01.950169 29202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:01.950341 29202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:01.958377 29202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:01.958513 29202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:01.958695 29202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:01.968761 29202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:01.970089 29202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:01.971521 29202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:01.984692 29208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:01.985030 29207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:01.985383 29210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:01.986347 29202 server_base.cc:1034] running on GCE node
I20260928 02:52:01.987103 29202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:01.988217 29202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:01.989434 29202 hybrid_clock.cc:648] HybridClock initialized: now 1790563921989356 us; error 69 us; skew 500 ppm
I20260928 02:52:01.989781 29202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:01 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563921, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:01.993062 29202 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:01.994679 29202 webserver.cc:460] Webserver started at http://127.27.229.193:39995/ using document root <none> and password file <none>
I20260928 02:52:01.995455 29202 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:01.995563 29202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:01.995836 29202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:01.998786 29202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "399b11e56f9f4ecb9f41bc490a71583c"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "172939b4fcea6c4a141087a0e7817290"
server_key_iv: "3419a3bf769c0d6b7d0070209759f462"
server_key_version: "encryptionkey@0"
I20260928 02:52:01.999646 29202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "399b11e56f9f4ecb9f41bc490a71583c"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "172939b4fcea6c4a141087a0e7817290"
server_key_iv: "3419a3bf769c0d6b7d0070209759f462"
server_key_version: "encryptionkey@0"
I20260928 02:52:02.005770 29202 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260928 02:52:02.010140 29217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.011968 29202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 02:52:02.012137 29202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "399b11e56f9f4ecb9f41bc490a71583c"
format_stamp: "Formatted at 2026-09-28 02:52:01 on dist-test-slave-s1kg"
server_key: "172939b4fcea6c4a141087a0e7817290"
server_key_iv: "3419a3bf769c0d6b7d0070209759f462"
server_key_version: "encryptionkey@0"
I20260928 02:52:02.012323 29202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:02.032572 29202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:02.037142 29202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:02.037400 29202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:02.038404 29202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:02.039780 29202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:02.039855 29202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.039942 29202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:02.039985 29202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.075508 29202 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:41411
I20260928 02:52:02.075613 29329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:41411 every 8 connection(s)
I20260928 02:52:02.076880 29202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:02.087375 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29202
I20260928 02:52:02.087625 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:02.088338 28567 external_mini_cluster.cc:1428] Setting key 3d03139ed6c046603e3aad8acdab58ba
Sep 28 02:52:02 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563921, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:02.102149 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:02.079894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:50025 (local address 127.27.229.254:33103)
0928 02:52:02.080430 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:02.080434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:02.081474 (+ 1040us) server_negotiation.cc:407] Connection header received
0928 02:52:02.083512 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:02.083519 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:02.083625 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:02.083749 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:02.085335 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.086086 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.088151 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.088844 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.089101 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:02.094529 (+ 5428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:02.094535 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:02.094538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:02.094575 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:02.098275 (+ 3700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:02.099379 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:02.099387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:02.099393 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:02.099563 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:02.100062 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:02.100069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:02.100075 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:02.100464 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:02.100624 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:02.101374 (+ 750us) server_negotiation.cc:299] Negotiation successful
0928 02:52:02.101654 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":306,"threads_started":1}
I20260928 02:52:02.103111 29333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:02.080441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33103 (local address 127.27.229.193:50025)
0928 02:52:02.081206 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:02.081246 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:02.083203 (+ 1957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:02.084014 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:02.084023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:02.084572 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:02.085148 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:02.085165 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.086267 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.086272 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:02.087940 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:02.087958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.089489 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.089501 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:02.089787 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:02.090653 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:02.090696 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:02.094184 (+ 3488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:02.098498 (+ 4314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:02.098506 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:02.098522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:02.099221 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:02.099706 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:02.099710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:02.099712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:02.099919 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:02.100589 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:02.100595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:02.100979 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 02:52:02.101212 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:52:02.101760 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":309,"threads_started":1}
I20260928 02:52:02.104774 29330 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33103
I20260928 02:52:02.105083 29330 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:02.105947 29330 heartbeater.cc:511] Master 127.27.229.254:33103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:52:02.108191 29141 ts_manager.cc:194] Registered new tserver with Master: 399b11e56f9f4ecb9f41bc490a71583c (127.27.229.193:41411)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:02.109922 29141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:50025
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:02.135442 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:02.475914 29336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:02.476274 29336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:02.476464 29336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:02.484004 29336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:02.484174 29336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:02.484367 29336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:02.494415 29336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:02.496006 29336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:02.497361 29336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:02.511360 29342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:02.511317 29341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:02.511457 29344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:02.512719 29336 server_base.cc:1034] running on GCE node
I20260928 02:52:02.513406 29336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:02.514441 29336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:02.515708 29336 hybrid_clock.cc:648] HybridClock initialized: now 1790563922515661 us; error 79 us; skew 500 ppm
I20260928 02:52:02.516054 29336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:02 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563922, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:02.519690 29336 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:02.521306 29336 webserver.cc:460] Webserver started at http://127.27.229.194:33485/ using document root <none> and password file <none>
I20260928 02:52:02.522058 29336 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:02.522189 29336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:02.522545 29336 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:02.525678 29336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "16c2d24ed91f419b925fb25d4f15f728"
format_stamp: "Formatted at 2026-09-28 02:52:02 on dist-test-slave-s1kg"
server_key: "868788d60b9ff8a8f104d0104c583d88"
server_key_iv: "b56fadbd5791c42a74c54a44645dd89b"
server_key_version: "encryptionkey@0"
I20260928 02:52:02.526527 29336 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "16c2d24ed91f419b925fb25d4f15f728"
format_stamp: "Formatted at 2026-09-28 02:52:02 on dist-test-slave-s1kg"
server_key: "868788d60b9ff8a8f104d0104c583d88"
server_key_iv: "b56fadbd5791c42a74c54a44645dd89b"
server_key_version: "encryptionkey@0"
I20260928 02:52:02.532070 29336 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:52:02.535867 29351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.537447 29336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:02.537645 29336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "16c2d24ed91f419b925fb25d4f15f728"
format_stamp: "Formatted at 2026-09-28 02:52:02 on dist-test-slave-s1kg"
server_key: "868788d60b9ff8a8f104d0104c583d88"
server_key_iv: "b56fadbd5791c42a74c54a44645dd89b"
server_key_version: "encryptionkey@0"
I20260928 02:52:02.537845 29336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:02.551888 29336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:02.556674 29336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:02.556996 29336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:02.558055 29336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:02.559624 29336 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:02.559698 29336 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.559787 29336 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:02.559834 29336 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:02.596846 29336 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:43301
I20260928 02:52:02.596897 29463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:43301 every 8 connection(s)
I20260928 02:52:02.598189 29336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:02.604300 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29336
I20260928 02:52:02.604513 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:02.604987 28567 external_mini_cluster.cc:1428] Setting key acada2fc21b5d282db2efa3a667217a2
Sep 28 02:52:02 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563922, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:02.620529 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:02.601360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:34141 (local address 127.27.229.254:33103)
0928 02:52:02.601584 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:02.601591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:02.602888 (+ 1297us) server_negotiation.cc:407] Connection header received
0928 02:52:02.605010 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:02.605015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:02.605108 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:02.605235 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:02.607243 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.608005 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.609688 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.610107 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.610248 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:02.614247 (+ 3999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:02.614252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:02.614255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:02.614289 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:02.617079 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:02.617946 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:02.617956 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:02.617958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:02.618075 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:02.618615 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:02.618619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:02.618621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:02.618967 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:02.619096 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:02.620117 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0928 02:52:02.620293 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:52:02.622129 29466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:02.601891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33103 (local address 127.27.229.194:34141)
0928 02:52:02.602692 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:02.602742 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:02.604701 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:02.605493 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:02.605502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:02.606227 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:02.607070 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:02.607091 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.608166 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.608171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:02.609529 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:02.609541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:02.610352 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:02.610361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:02.610592 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:02.611384 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:02.611408 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:02.613987 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:02.617238 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:02.617248 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:02.617268 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:02.617809 (+ 541us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:02.618185 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:02.618192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:02.618199 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:02.618469 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:02.619106 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:02.619117 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:02.619668 (+ 551us) client_negotiation.cc:769] Sending connection context
0928 02:52:02.619997 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 02:52:02.620613 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:52:02.623945 29464 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33103
I20260928 02:52:02.624310 29464 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:52:02.625303 29464 heartbeater.cc:511] Master 127.27.229.254:33103 requested a full tablet report, sending...
I20260928 02:52:02.626968 29141 ts_manager.cc:194] Registered new tserver with Master: 16c2d24ed91f419b925fb25d4f15f728 (127.27.229.194:43301)
I20260928 02:52:02.627825 29141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:34141
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:02.650624 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:02.980450 29469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:02.980764 29469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:02.980930 29469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:02.988612 29469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:02.988734 29469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:02.988921 29469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:02.999185 29469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:03.000545 29469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:03.001927 29469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:03.016321 29474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:03.017216 29477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:03.017324 29475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:03.017874 29469 server_base.cc:1034] running on GCE node
I20260928 02:52:03.018534 29469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:03.019610 29469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:03.020871 29469 hybrid_clock.cc:648] HybridClock initialized: now 1790563923020812 us; error 52 us; skew 500 ppm
I20260928 02:52:03.021257 29469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:03.025390 29469 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:03.027117 29469 webserver.cc:460] Webserver started at http://127.27.229.195:38105/ using document root <none> and password file <none>
I20260928 02:52:03.027844 29469 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:03.027992 29469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:03.028301 29469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:03.031397 29469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "97c956f1ca6546169cad4983b4eef4f0"
format_stamp: "Formatted at 2026-09-28 02:52:03 on dist-test-slave-s1kg"
server_key: "f4d853d5be433ad8d6ab3840238b27f5"
server_key_iv: "3f5569616e118fa5256779138729ef73"
server_key_version: "encryptionkey@0"
I20260928 02:52:03.032260 29469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "97c956f1ca6546169cad4983b4eef4f0"
format_stamp: "Formatted at 2026-09-28 02:52:03 on dist-test-slave-s1kg"
server_key: "f4d853d5be433ad8d6ab3840238b27f5"
server_key_iv: "3f5569616e118fa5256779138729ef73"
server_key_version: "encryptionkey@0"
I20260928 02:52:03.037606 29469 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:03.041309 29484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:03.042783 29469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:52:03.042996 29469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "97c956f1ca6546169cad4983b4eef4f0"
format_stamp: "Formatted at 2026-09-28 02:52:03 on dist-test-slave-s1kg"
server_key: "f4d853d5be433ad8d6ab3840238b27f5"
server_key_iv: "3f5569616e118fa5256779138729ef73"
server_key_version: "encryptionkey@0"
I20260928 02:52:03.043231 29469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:03.061830 29469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:03.066165 29469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:03.066486 29469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:03.067534 29469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:03.068922 29469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:03.069026 29469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:03.069185 29469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:03.069257 29469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:03.103482 29469 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:35011
I20260928 02:52:03.103521 29596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:35011 every 8 connection(s)
I20260928 02:52:03.104781 29469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:03.109119 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29469
I20260928 02:52:03.109305 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:03.109735 28567 external_mini_cluster.cc:1428] Setting key def279ff946910f2fc81126a09a10ddf
I20260928 02:52:03.113161 29330 heartbeater.cc:503] Master 127.27.229.254:33103 was elected leader, sending a full tablet report...
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:03.126943 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.107663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46119 (local address 127.27.229.254:33103)
0928 02:52:03.107879 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.107886 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.109209 (+ 1323us) server_negotiation.cc:407] Connection header received
0928 02:52:03.111341 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.111346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.111434 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.111553 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.113341 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.114511 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.116299 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.116720 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.116827 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.120518 (+ 3691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.120524 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.120527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.120560 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.123437 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.124456 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.124461 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.124464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.124637 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.125129 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.125135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.125138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.125449 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.125566 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.126391 (+ 825us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.126606 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 02:52:03.127911 29599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.108275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33103 (local address 127.27.229.195:46119)
0928 02:52:03.109053 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.109091 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.111022 (+ 1931us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.111800 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.111808 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.112314 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:03.113083 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.113109 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.114677 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.114683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.116138 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.116150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.117025 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.117033 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.117267 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.117991 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:03.118019 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:03.120261 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:03.123595 (+ 3334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:03.123605 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:03.123649 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:03.124300 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:03.124751 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:03.124757 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:03.124761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:03.124959 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:03.125641 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:03.125648 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:03.126005 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.126250 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.126653 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":266,"threads_started":1}
I20260928 02:52:03.129359 29597 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33103
I20260928 02:52:03.129658 29597 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:03.130364 29597 heartbeater.cc:511] Master 127.27.229.254:33103 requested a full tablet report, sending...
I20260928 02:52:03.131866 29141 ts_manager.cc:194] Registered new tserver with Master: 97c956f1ca6546169cad4983b4eef4f0 (127.27.229.195:35011)
I20260928 02:52:03.132627 29141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:46119
I20260928 02:52:03.137701 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:52:03.157073 29332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.144137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.27.229.254:33103)
0928 02:52:03.144364 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.144369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.144918 (+ 549us) server_negotiation.cc:407] Connection header received
0928 02:52:03.145038 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.145042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.145106 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.145196 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.146889 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.147815 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.149144 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.149612 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.149719 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.152020 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.152025 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.152028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.152060 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.154692 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.155408 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.155412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.155415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.155586 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.155890 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.155893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.155895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.156192 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.156364 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.156681 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.156837 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 02:52:03.163694 29141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57236:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:03.170456 29141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:03.195654 29609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.183354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:43301 (local address 127.0.0.1:40188)
0928 02:52:03.184266 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.184322 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.184540 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.185869 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.185881 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.185918 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:03.186377 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.186415 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.189131 (+ 2716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.189140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.192382 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.192415 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.194563 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.194570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.194817 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.194900 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.195024 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.195227 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":3,"thread_start_us":1181,"threads_started":1}
I20260928 02:52:03.195906 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.183490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.27.229.194:43301)
0928 02:52:03.185150 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.185158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.185187 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:52:03.185283 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.185290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.185582 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.186648 (+ 1066us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:03.186949 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.188935 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.192683 (+ 3748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.194281 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.195188 (+ 907us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.195363 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.195431 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.195598 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1516,"thread_start_us":1424,"threads_started":1}
I20260928 02:52:03.200008 29399 tablet_service.cc:1467] Processing CreateTablet for tablet 18affaa011f143c79c5ed92764d1fb31 (DEFAULT_TABLE table=test-table [id=08b52e8aab484f9ba9739d47f368384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:03.201438 29399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18affaa011f143c79c5ed92764d1fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:03.201875 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.184382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47666 (local address 127.27.229.193:41411)
0928 02:52:03.193411 (+ 9029us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.193422 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.193913 (+ 491us) server_negotiation.cc:407] Connection header received
0928 02:52:03.194020 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.194028 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.194301 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.194554 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:03.195321 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.196913 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.199324 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.200790 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.200989 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.201153 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.201383 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.201565 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8846,"thread_start_us":8671,"threads_started":1}
I20260928 02:52:03.202123 29611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.183613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41411 (local address 127.0.0.1:47666)
0928 02:52:03.193640 (+ 10027us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.193662 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.193784 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.194592 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.194597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.194621 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:03.195131 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.195143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.197069 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.197075 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.199158 (+ 2083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.199176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.201012 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.201018 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.201175 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.201210 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.201298 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.201427 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9910,"mutex_wait_us":1,"thread_start_us":10321,"threads_started":1}
I20260928 02:52:03.203440 29613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.185490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:35011 (local address 127.0.0.1:40072)
0928 02:52:03.194677 (+ 9187us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.194704 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.194847 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.195801 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.195806 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.195829 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:03.196106 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.196115 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.198499 (+ 2384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.198507 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.200467 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.200482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.202678 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.202685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.202885 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.202916 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.203015 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.203142 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9031,"thread_start_us":8810,"threads_started":1}
I20260928 02:52:03.204191 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.186113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40072 (local address 127.27.229.195:35011)
0928 02:52:03.193896 (+ 7783us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.193907 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.194996 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 02:52:03.195105 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.195115 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.195350 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.195551 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:03.196287 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.198356 (+ 2069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.200661 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.202354 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.202555 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.202727 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.203671 (+ 944us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.203837 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7606,"thread_start_us":7606,"threads_started":1}
I20260928 02:52:03.206646 29265 tablet_service.cc:1467] Processing CreateTablet for tablet 18affaa011f143c79c5ed92764d1fb31 (DEFAULT_TABLE table=test-table [id=08b52e8aab484f9ba9739d47f368384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:03.207221 29532 tablet_service.cc:1467] Processing CreateTablet for tablet 18affaa011f143c79c5ed92764d1fb31 (DEFAULT_TABLE table=test-table [id=08b52e8aab484f9ba9739d47f368384d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:03.208137 29265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18affaa011f143c79c5ed92764d1fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:03.208592 29532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18affaa011f143c79c5ed92764d1fb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:03.217415 29615 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Bootstrap starting.
I20260928 02:52:03.221026 29616 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Bootstrap starting.
I20260928 02:52:03.221313 29615 tablet_bootstrap.cc:654] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:03.222529 29615 log.cc:824] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:03.223634 29617 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Bootstrap starting.
I20260928 02:52:03.225005 29616 tablet_bootstrap.cc:654] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:03.226209 29616 log.cc:824] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:03.227057 29617 tablet_bootstrap.cc:654] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:03.227710 29615 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: No bootstrap required, opened a new log
I20260928 02:52:03.227985 29615 ts_tablet_manager.cc:1397] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260928 02:52:03.228190 29617 log.cc:824] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:03.229200 29616 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: No bootstrap required, opened a new log
I20260928 02:52:03.229461 29616 ts_tablet_manager.cc:1397] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:52:03.230611 29617 tablet_bootstrap.cc:492] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: No bootstrap required, opened a new log
I20260928 02:52:03.230914 29617 ts_tablet_manager.cc:1397] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 02:52:03.238941 29616 raft_consensus.cc:348] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.239444 29616 raft_consensus.cc:374] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:03.239539 29616 raft_consensus.cc:729] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c956f1ca6546169cad4983b4eef4f0, State: Initialized, Role: FOLLOWER
I20260928 02:52:03.239425 29615 raft_consensus.cc:348] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.239957 29615 raft_consensus.cc:374] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:03.240060 29615 raft_consensus.cc:729] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16c2d24ed91f419b925fb25d4f15f728, State: Initialized, Role: FOLLOWER
I20260928 02:52:03.240229 29616 consensus_queue.cc:260] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.240636 29615 consensus_queue.cc:260] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.240944 29617 raft_consensus.cc:348] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.241508 29597 heartbeater.cc:503] Master 127.27.229.254:33103 was elected leader, sending a full tablet report...
I20260928 02:52:03.241546 29617 raft_consensus.cc:374] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:03.241663 29617 raft_consensus.cc:729] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 399b11e56f9f4ecb9f41bc490a71583c, State: Initialized, Role: FOLLOWER
I20260928 02:52:03.242225 29616 ts_tablet_manager.cc:1428] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 02:52:03.242478 29617 consensus_queue.cc:260] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.243510 29464 heartbeater.cc:503] Master 127.27.229.254:33103 was elected leader, sending a full tablet report...
I20260928 02:52:03.244484 29615 ts_tablet_manager.cc:1428] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260928 02:52:03.246255 29617 ts_tablet_manager.cc:1428] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 02:52:03.303781 29623 raft_consensus.cc:482] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:03.304108 29623 raft_consensus.cc:504] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.305994 29623 leader_election.cc:288] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16c2d24ed91f419b925fb25d4f15f728 (127.27.229.194:43301), 97c956f1ca6546169cad4983b4eef4f0 (127.27.229.195:35011)
I20260928 02:52:03.313956 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.306479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:33333 (local address 127.27.229.194:43301)
0928 02:52:03.306774 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.306784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.307362 (+ 578us) server_negotiation.cc:407] Connection header received
0928 02:52:03.307571 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.307579 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.307672 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.307768 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:03.308541 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.309335 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.311759 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.313046 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.313175 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.313228 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.313647 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.313745 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:52:03.314101 29624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.306364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:43301 (local address 127.27.229.193:33333)
0928 02:52:03.307212 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.307228 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.307411 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.307855 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.307859 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.307883 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:03.308377 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.308391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.309535 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.309543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.311558 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.311570 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.313266 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.313273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.313452 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.313492 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.313597 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.313735 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":501,"threads_started":1}
I20260928 02:52:03.315219 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.307535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:40071 (local address 127.27.229.195:35011)
0928 02:52:03.307756 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.307762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.308184 (+ 422us) server_negotiation.cc:407] Connection header received
0928 02:52:03.308300 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.308306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.308399 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.308501 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:03.309484 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.310605 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.313003 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.314318 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.314492 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.314572 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.314945 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.315047 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:52:03.315389 29625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.307368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:35011 (local address 127.27.229.193:40071)
0928 02:52:03.308083 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:03.308097 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:03.308199 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:03.308881 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:03.308895 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:03.308920 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:03.309302 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.309314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.310800 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.310804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.312793 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:03.312809 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.314557 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.314563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:03.314736 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.314777 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:52:03.314925 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:52:03.315057 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":277,"threads_started":1}
I20260928 02:52:03.315259 29419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18affaa011f143c79c5ed92764d1fb31" candidate_uuid: "399b11e56f9f4ecb9f41bc490a71583c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c2d24ed91f419b925fb25d4f15f728" is_pre_election: true
I20260928 02:52:03.315929 29419 raft_consensus.cc:2454] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399b11e56f9f4ecb9f41bc490a71583c in term 0.
I20260928 02:52:03.316352 29552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18affaa011f143c79c5ed92764d1fb31" candidate_uuid: "399b11e56f9f4ecb9f41bc490a71583c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c956f1ca6546169cad4983b4eef4f0" is_pre_election: true
I20260928 02:52:03.316690 29218 leader_election.cc:302] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c2d24ed91f419b925fb25d4f15f728, 399b11e56f9f4ecb9f41bc490a71583c; no voters:
I20260928 02:52:03.317001 29552 raft_consensus.cc:2454] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 399b11e56f9f4ecb9f41bc490a71583c in term 0.
I20260928 02:52:03.317098 29623 raft_consensus.cc:2781] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:03.317276 29623 raft_consensus.cc:482] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:03.317384 29623 raft_consensus.cc:3037] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:03.319844 29623 raft_consensus.cc:504] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
I20260928 02:52:03.320436 29623 leader_election.cc:288] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [CANDIDATE]: Term 1 election: Requested vote from peers 16c2d24ed91f419b925fb25d4f15f728 (127.27.229.194:43301), 97c956f1ca6546169cad4983b4eef4f0 (127.27.229.195:35011)
I20260928 02:52:03.321070 29419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18affaa011f143c79c5ed92764d1fb31" candidate_uuid: "399b11e56f9f4ecb9f41bc490a71583c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16c2d24ed91f419b925fb25d4f15f728"
I20260928 02:52:03.321094 29552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18affaa011f143c79c5ed92764d1fb31" candidate_uuid: "399b11e56f9f4ecb9f41bc490a71583c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c956f1ca6546169cad4983b4eef4f0"
I20260928 02:52:03.321282 29419 raft_consensus.cc:3037] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:03.321306 29552 raft_consensus.cc:3037] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:03.323719 29552 raft_consensus.cc:2454] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399b11e56f9f4ecb9f41bc490a71583c in term 1.
I20260928 02:52:03.323724 29419 raft_consensus.cc:2454] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 399b11e56f9f4ecb9f41bc490a71583c in term 1.
I20260928 02:52:03.324152 29218 leader_election.cc:302] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16c2d24ed91f419b925fb25d4f15f728, 399b11e56f9f4ecb9f41bc490a71583c; no voters:
I20260928 02:52:03.324456 29623 raft_consensus.cc:2781] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:03.324900 29623 raft_consensus.cc:686] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [term 1 LEADER]: Becoming Leader. State: Replica: 399b11e56f9f4ecb9f41bc490a71583c, State: Running, Role: LEADER
I20260928 02:52:03.325395 29623 consensus_queue.cc:237] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } }
W20260928 02:52:03.329380 29331 tablet.cc:2369] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:03.330145 29139 catalog_manager.cc:5495] T 18affaa011f143c79c5ed92764d1fb31 P 399b11e56f9f4ecb9f41bc490a71583c reported cstate change: term changed from 0 to 1, leader changed from <none> to 399b11e56f9f4ecb9f41bc490a71583c (127.27.229.193). New cstate: current_term: 1 leader_uuid: "399b11e56f9f4ecb9f41bc490a71583c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16c2d24ed91f419b925fb25d4f15f728" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "399b11e56f9f4ecb9f41bc490a71583c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97c956f1ca6546169cad4983b4eef4f0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 35011 } health_report { overall_health: UNKNOWN } } }
W20260928 02:52:03.350386 29465 tablet.cc:2369] T 18affaa011f143c79c5ed92764d1fb31 P 16c2d24ed91f419b925fb25d4f15f728: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
W20260928 02:52:03.356745 29598 tablet.cc:2369] T 18affaa011f143c79c5ed92764d1fb31 P 97c956f1ca6546169cad4983b4eef4f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:03.358521 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.342399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.27.229.193:41411)
0928 02:52:03.342763 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.342772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.343114 (+ 342us) server_negotiation.cc:407] Connection header received
0928 02:52:03.343350 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.343354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.343431 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.343537 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.345110 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.345836 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.347373 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.347793 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.347896 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.352081 (+ 4185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.352096 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.352111 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.352148 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.355324 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.356139 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.356148 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.356150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.356372 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.356840 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.356843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.356845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.357271 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.357549 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.357940 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.358183 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:03.408950 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.393943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47688 (local address 127.27.229.193:41411)
0928 02:52:03.394213 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.394222 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.394583 (+ 361us) server_negotiation.cc:407] Connection header received
0928 02:52:03.394818 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.394822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.394968 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.395080 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.396526 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.397251 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.398554 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.399041 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.399191 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.403450 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.403455 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.403459 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.403496 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.406280 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.407026 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.407030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.407033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.407164 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.407531 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.407534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.407535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.407850 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.408060 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.408369 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.408585 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:03.454146 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.439826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47704 (local address 127.27.229.193:41411)
0928 02:52:03.440096 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.440103 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.440332 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:52:03.440535 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.440539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.440655 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.440776 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.442112 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.443004 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.444219 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.444627 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.444721 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.448859 (+ 4138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.448864 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.448867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.448902 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.451776 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.452496 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.452500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.452502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.452633 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.453003 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.453007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.453009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.453276 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.453500 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.453730 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.453887 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 02:52:03.456082 29245 tablet_service.cc:2908] Rejecting scan request for tablet 18affaa011f143c79c5ed92764d1fb31: Uninitialized: clean time has not yet been initialized
I20260928 02:52:03.459326 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29202
I20260928 02:52:03.475261 29612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.460313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47714 (local address 127.27.229.193:41411)
0928 02:52:03.460475 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.460479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.463368 (+ 2889us) server_negotiation.cc:407] Connection header received
0928 02:52:03.463581 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.463587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.463656 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.463769 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.465006 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.465714 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.466969 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.467375 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.467518 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.469822 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.469830 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.469840 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.469872 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.472544 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.473249 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.473256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.473259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.473387 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.473770 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.473775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.473778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.474056 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.474328 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.474626 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.474919 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:52:03.475899 29325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:03.569880 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29202
I20260928 02:52:03.587097 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29336
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:03.603734 29610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.588084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40190 (local address 127.27.229.194:43301)
0928 02:52:03.588305 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.588316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.588332 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:52:03.588383 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.588388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.588454 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.588567 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.589765 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.590509 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.591913 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.592322 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.592456 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.596575 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.596587 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.596603 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.596649 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.600641 (+ 3992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.601441 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.601454 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.601459 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.601617 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.602015 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.602020 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.602043 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.602536 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.602860 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.603143 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.603378 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:52:03.604527 29459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:03.700070 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29336
I20260928 02:52:03.718386 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29469
Sep 28 02:52:03 dist-test-slave-s1kg krb5kdc[29096](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:03.735040 29614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:03.719296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.27.229.195:35011)
0928 02:52:03.719558 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:03.719564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:03.719586 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:52:03.719649 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:03.719654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:03.719714 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:03.719818 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:03.721124 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.721900 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.723347 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:03.723870 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:03.724014 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:03.728853 (+ 4839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:03.728873 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:03.728891 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:03.728935 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:03.732092 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.732864 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.732872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.732874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.733004 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:03.733412 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:03.733417 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:03.733422 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:03.733808 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:03.734119 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:03.734416 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 02:52:03.734700 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:03.735721 29592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:03.833386 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29469
I20260928 02:52:03.849946 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29109
I20260928 02:52:03.851331 29171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:03.965749 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29109
2026-09-28T02:52:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4271 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:06 dist-test-slave-s1kg krb5kdc[29645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:06 dist-test-slave-s1kg krb5kdc[29645](info): set up 2 sockets
Sep 28 02:52:06 dist-test-slave-s1kg krb5kdc[29645](info): commencing operation
krb5kdc: starting...
W20260928 02:52:07.133589 28567 mini_kdc.cc:121] Time spent starting KDC: real 3.124s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 02:52:07.260744 28567 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.127s user 0.000s sys 0.004s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:07 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:07.428083 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42665
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:42665
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:07.766717 29658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:07.767097 29658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:07.767275 29658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:07.774459 29658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:07.774617 29658 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:52:07.774709 29658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:07.774803 29658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:07.774919 29658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:07.775007 29658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:07.784843 29658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:42665
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:07.786248 29658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:07.787642 29658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:07.799826 29663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:07.799973 29664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:07.799938 29666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:07.801072 29658 server_base.cc:1034] running on GCE node
I20260928 02:52:07.801923 29658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:07.803556 29658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:07.804888 29658 hybrid_clock.cc:648] HybridClock initialized: now 1790563927804799 us; error 40 us; skew 500 ppm
I20260928 02:52:07.805287 29658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:07 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563927, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:07.814580 29658 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:07.816431 29658 webserver.cc:460] Webserver started at http://127.27.229.254:35785/ using document root <none> and password file <none>
I20260928 02:52:07.817209 29658 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:07.817341 29658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:07.817672 29658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:07.820750 29658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "eaa9d888458c481187c1a12f2555e79f"
format_stamp: "Formatted at 2026-09-28 02:52:07 on dist-test-slave-s1kg"
server_key: "3ad1c57dfb3616c38b997681a308c529"
server_key_iv: "f9446a3af259a12ef222af908aec5879"
server_key_version: "encryptionkey@0"
I20260928 02:52:07.821528 29658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "eaa9d888458c481187c1a12f2555e79f"
format_stamp: "Formatted at 2026-09-28 02:52:07 on dist-test-slave-s1kg"
server_key: "3ad1c57dfb3616c38b997681a308c529"
server_key_iv: "f9446a3af259a12ef222af908aec5879"
server_key_version: "encryptionkey@0"
I20260928 02:52:07.827086 29658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:52:07.830803 29673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:07.832351 29658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:07.832556 29658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "eaa9d888458c481187c1a12f2555e79f"
format_stamp: "Formatted at 2026-09-28 02:52:07 on dist-test-slave-s1kg"
server_key: "3ad1c57dfb3616c38b997681a308c529"
server_key_iv: "f9446a3af259a12ef222af908aec5879"
server_key_version: "encryptionkey@0"
I20260928 02:52:07.832834 29658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:07.861251 29658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:07.866155 29658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:07.866489 29658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:07.892061 29658 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:42665
I20260928 02:52:07.892024 29734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:42665 every 8 connection(s)
I20260928 02:52:07.893757 29658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:07.897091 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29658
I20260928 02:52:07.897461 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:07.898494 28567 external_mini_cluster.cc:1428] Setting key 10fbef57d11c3ce9a1b35cab8922ef03
I20260928 02:52:07.899080 29735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:07.917922 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Bootstrap starting.
Sep 28 02:52:07 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:07.922559 29735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:07.924059 29735 log.cc:824] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:07.927539 29735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: No bootstrap required, opened a new log
I20260928 02:52:07.930514 29738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:07.901141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48274 (local address 127.27.229.254:42665)
0928 02:52:07.902496 (+ 1355us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:07.902727 (+ 231us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:07.902781 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:52:07.906337 (+ 3556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:07.906375 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:07.907010 (+ 635us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:07.907510 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:07.910121 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:07.912341 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:07.914279 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:07.915579 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:07.915786 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:07.920173 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:07.920184 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:07.920229 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:07.920283 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:07.924917 (+ 4634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:07.925856 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:07.925865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:07.925874 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:07.926073 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:07.926511 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:07.926516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:07.926519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:07.927204 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:07.928417 (+ 1213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:07.928685 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:52:07.929104 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":820,"threads_started":1}
I20260928 02:52:07.938513 29735 raft_consensus.cc:348] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } }
I20260928 02:52:07.938982 29735 raft_consensus.cc:374] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:07.939085 29735 raft_consensus.cc:729] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eaa9d888458c481187c1a12f2555e79f, State: Initialized, Role: FOLLOWER
I20260928 02:52:07.939867 29735 consensus_queue.cc:260] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } }
I20260928 02:52:07.940194 29735 raft_consensus.cc:388] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:07.940333 29735 raft_consensus.cc:482] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:07.940518 29735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:07.942723 29735 raft_consensus.cc:504] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } }
I20260928 02:52:07.943351 29735 leader_election.cc:302] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eaa9d888458c481187c1a12f2555e79f; no voters:
I20260928 02:52:07.944033 29735 leader_election.cc:288] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:07.944191 29740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:07.944958 29740 raft_consensus.cc:686] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [term 1 LEADER]: Becoming Leader. State: Replica: eaa9d888458c481187c1a12f2555e79f, State: Running, Role: LEADER
I20260928 02:52:07.945528 29740 consensus_queue.cc:237] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } }
I20260928 02:52:07.946238 29735 sys_catalog.cc:564] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:07.948606 29742 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eaa9d888458c481187c1a12f2555e79f. Latest consensus state: current_term: 1 leader_uuid: "eaa9d888458c481187c1a12f2555e79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } } }
I20260928 02:52:07.949013 29742 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:07.949628 29741 sys_catalog.cc:455] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eaa9d888458c481187c1a12f2555e79f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eaa9d888458c481187c1a12f2555e79f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42665 } } }
I20260928 02:52:07.949942 29741 sys_catalog.cc:458] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:07.951391 29747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:07.958559 29747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:07.969964 29747 catalog_manager.cc:1347] Generated new cluster ID: a1d627b69b2d41b49a5b644418e4a82d
I20260928 02:52:07.970095 29747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:07.989920 29747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:07.991936 29747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:08.003465 29747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Generated new TSK 0
I20260928 02:52:08.004680 29747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:08.053979 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42665
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:08.390182 29761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:08.390475 29761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:08.390580 29761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:08.398119 29761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:08.398238 29761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:08.398320 29761 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:08.398401 29761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:08.408456 29761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:08.409785 29761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:08.411166 29761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:08.424957 29766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:08.424871 29769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:08.424914 29767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:08.426016 29761 server_base.cc:1034] running on GCE node
I20260928 02:52:08.426749 29761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:08.427772 29761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:08.429064 29761 hybrid_clock.cc:648] HybridClock initialized: now 1790563928428986 us; error 41 us; skew 500 ppm
I20260928 02:52:08.429442 29761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:08 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:08.432924 29761 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:08.434473 29761 webserver.cc:460] Webserver started at http://127.27.229.193:45917/ using document root <none> and password file <none>
I20260928 02:52:08.435258 29761 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:08.435372 29761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:08.435721 29761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:08.438948 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "38ecea8616dd4af886ac208edd8b9909"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "93122843d48f8526ded1abe1336e5ee4"
server_key_iv: "5c17010f65f16b588f9898667371b051"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.439824 29761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "38ecea8616dd4af886ac208edd8b9909"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "93122843d48f8526ded1abe1336e5ee4"
server_key_iv: "5c17010f65f16b588f9898667371b051"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.445129 29761 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:08.448778 29776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:08.450119 29761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:08.450300 29761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "38ecea8616dd4af886ac208edd8b9909"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "93122843d48f8526ded1abe1336e5ee4"
server_key_iv: "5c17010f65f16b588f9898667371b051"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.450529 29761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:08.471932 29761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:08.476652 29761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:08.476955 29761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:08.480060 29761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:08.480201 29761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:08.480346 29761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:08.480424 29761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:52:08 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:08.507299 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:08.480764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48659 (local address 127.27.229.254:42665)
0928 02:52:08.481369 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:08.481373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:08.482360 (+ 987us) server_negotiation.cc:407] Connection header received
0928 02:52:08.484478 (+ 2118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:08.484483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:08.484591 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:08.484710 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:08.487094 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.488389 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.491230 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.492013 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.492173 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:08.497894 (+ 5721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:08.497905 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:08.497913 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:08.497971 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:08.502280 (+ 4309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:08.503617 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:08.503626 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:08.503640 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:08.503834 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:08.504551 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:08.504558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:08.504561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:08.504894 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:08.505078 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:08.506388 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0928 02:52:08.506674 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":393,"threads_started":1}
I20260928 02:52:08.509255 29786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:08.481337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.193:48659)
0928 02:52:08.482183 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:08.482229 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:08.484187 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:08.484967 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:08.485002 (+ 35us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:08.485912 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:08.486807 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:08.486883 (+ 76us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.488619 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.488626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:08.491047 (+ 2421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:08.491068 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.492608 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.492622 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:08.492914 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:08.493867 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:08.493905 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:08.497564 (+ 3659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:08.502534 (+ 4970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:08.502550 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:08.502577 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:08.503415 (+ 838us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:08.503978 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:08.503986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:08.503992 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:08.504375 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:08.505056 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:08.505069 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:08.505737 (+ 668us) client_negotiation.cc:769] Sending connection context
0928 02:52:08.506173 (+ 436us) client_negotiation.cc:240] Negotiation successful
0928 02:52:08.507162 (+ 989us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":343,"threads_started":1}
I20260928 02:52:08.529278 29761 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:41347
I20260928 02:52:08.529318 29896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:41347 every 8 connection(s)
I20260928 02:52:08.530750 29761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:08.533543 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29761
I20260928 02:52:08.533762 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:08.534060 29897 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42665
I20260928 02:52:08.534312 29897 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:08.534559 28567 external_mini_cluster.cc:1428] Setting key b9380269fea5af0cf4fb81cb194474ce
I20260928 02:52:08.535202 29897 heartbeater.cc:511] Master 127.27.229.254:42665 requested a full tablet report, sending...
I20260928 02:52:08.535722 29872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:08.517038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.193:55293)
0928 02:52:08.517908 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:08.517934 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:08.518127 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:08.518494 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:08.518501 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:08.519242 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:08.519966 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:08.519985 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.521312 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.521321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:08.523997 (+ 2676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:08.524025 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.525088 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.525097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:08.525274 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:08.526105 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:08.526123 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:08.526997 (+ 874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:08.530456 (+ 3459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:08.530461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:08.530465 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:08.531061 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:08.531550 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:08.531557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:08.531562 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:08.531779 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:08.532633 (+ 854us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:08.532637 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:08.532850 (+ 213us) client_negotiation.cc:769] Sending connection context
0928 02:52:08.534941 (+ 2091us) client_negotiation.cc:240] Negotiation successful
0928 02:52:08.535368 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":1,"thread_start_us":364,"threads_started":1}
I20260928 02:52:08.535723 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:08.516996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:55293 (local address 127.27.229.254:42665)
0928 02:52:08.517195 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:08.517202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:08.518066 (+ 864us) server_negotiation.cc:407] Connection header received
0928 02:52:08.518253 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:08.518257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:08.518321 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:08.518409 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:08.520180 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.521134 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.524216 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:08.524824 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:08.525046 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:08.527210 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:08.527215 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:08.527219 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:08.527251 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:08.530307 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:08.531202 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:08.531206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:08.531208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:08.531400 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:08.531934 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:08.531940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:08.531943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:08.532335 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:08.532528 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:08.535082 (+ 2554us) server_negotiation.cc:299] Negotiation successful
0928 02:52:08.535363 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:08.537708 29690 ts_manager.cc:194] Registered new tserver with Master: 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347)
I20260928 02:52:08.539685 29690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:48659
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:08.578529 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42665
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:08.913173 29901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:08.913530 29901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:08.913712 29901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:08.921551 29901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:08.921767 29901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:08.921939 29901 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:08.922091 29901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:08.933259 29901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:08.934787 29901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:08.936380 29901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:08.951033 29906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:08.950950 29907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:08.950961 29909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:08.951999 29901 server_base.cc:1034] running on GCE node
I20260928 02:52:08.952728 29901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:08.953843 29901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:08.955158 29901 hybrid_clock.cc:648] HybridClock initialized: now 1790563928955068 us; error 57 us; skew 500 ppm
I20260928 02:52:08.955657 29901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:08 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:08.959297 29901 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:08.960877 29901 webserver.cc:460] Webserver started at http://127.27.229.194:36865/ using document root <none> and password file <none>
I20260928 02:52:08.961627 29901 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:08.961716 29901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:08.962028 29901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:08.965379 29901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "abdbcbca5d33499fa20a0543ef93be5e"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "acb62384f75204e3320cc18783a7add3"
server_key_iv: "d97c74ff0cd483ee013b4cc2a3413a6c"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.966235 29901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "abdbcbca5d33499fa20a0543ef93be5e"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "acb62384f75204e3320cc18783a7add3"
server_key_iv: "d97c74ff0cd483ee013b4cc2a3413a6c"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.972276 29901 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 02:52:08.976295 29916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:08.977892 29901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:08.978104 29901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "abdbcbca5d33499fa20a0543ef93be5e"
format_stamp: "Formatted at 2026-09-28 02:52:08 on dist-test-slave-s1kg"
server_key: "acb62384f75204e3320cc18783a7add3"
server_key_iv: "d97c74ff0cd483ee013b4cc2a3413a6c"
server_key_version: "encryptionkey@0"
I20260928 02:52:08.978368 29901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:08.999809 29901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:09.004699 29901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:09.004990 29901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:09.007936 29901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:09.008067 29901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:09.008227 29901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:09.008304 29901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:52:09 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563928, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:09.031440 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.008619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:41669 (local address 127.27.229.254:42665)
0928 02:52:09.008947 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.008954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.010582 (+ 1628us) server_negotiation.cc:407] Connection header received
0928 02:52:09.012815 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.012820 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.012914 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.013039 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:09.015477 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.016813 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.018635 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.019532 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.019693 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.024575 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:09.024584 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:09.024590 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:09.024642 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:09.028116 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.028969 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.028975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.028980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.029112 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.029580 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.029587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.029594 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.029885 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:09.030048 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.030810 (+ 762us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.031075 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260928 02:52:09.032955 29925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.009382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.194:41669)
0928 02:52:09.010392 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.010453 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.012507 (+ 2054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.013304 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.013313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.014210 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:09.015225 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.015248 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.017047 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.017051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.018468 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.018481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.019830 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.019839 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.020062 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.020917 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:09.020966 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:09.024276 (+ 3310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:09.028315 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.028325 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.028347 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.028816 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.029232 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.029236 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.029240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.029451 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.030025 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:09.030033 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:09.030434 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.030713 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.031333 (+ 620us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":458,"threads_started":1}
I20260928 02:52:09.059916 29901 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:46761
I20260928 02:52:09.059952 30035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:46761 every 8 connection(s)
I20260928 02:52:09.060285 29785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.042035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:51033 (local address 127.27.229.254:42665)
0928 02:52:09.042282 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.042290 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.043605 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 02:52:09.043827 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.043831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.043911 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.044130 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:09.045124 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.046364 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.049136 (+ 2772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.049779 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.049886 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.052947 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:09.052956 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:09.052964 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:09.053011 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:09.056453 (+ 3442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.057611 (+ 1158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.057619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.057626 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.057837 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.058253 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.058257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.058259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.058760 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:09.058943 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.059606 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.059890 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:52:09.060494 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.042054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.194:51033)
0928 02:52:09.043439 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.043466 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.043678 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.044166 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.044171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.044676 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:09.044960 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.044970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.046544 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.046557 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.048937 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.048961 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.050007 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.050014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.050236 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.051112 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:09.051151 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:09.052668 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:09.056636 (+ 3968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.056649 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.056656 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.057436 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.057956 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.057961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.057964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.058135 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.058945 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:09.058954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:09.059251 (+ 297us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.059448 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.059913 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"thread_start_us":985,"threads_started":1}
I20260928 02:52:09.061793 29901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:09.064389 30036 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42665
I20260928 02:52:09.064869 30036 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:09.065569 30036 heartbeater.cc:511] Master 127.27.229.254:42665 requested a full tablet report, sending...
I20260928 02:52:09.067163 29690 ts_manager.cc:194] Registered new tserver with Master: abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761)
I20260928 02:52:09.068004 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 29901
I20260928 02:52:09.068027 29690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:41669
I20260928 02:52:09.068183 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:09.068763 28567 external_mini_cluster.cc:1428] Setting key 869c09aedd782ec91826ebada98d87f9
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:09.111207 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42665
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:09.451150 30040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:09.451438 30040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:09.451581 30040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:09.459091 30040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:09.459206 30040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:09.459372 30040 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:09.459465 30040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:09.469506 30040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:42665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:09.470858 30040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:09.472307 30040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:09.486016 30046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:09.486115 30048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:09.486116 30045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:09.486779 30040 server_base.cc:1034] running on GCE node
I20260928 02:52:09.487540 30040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:09.488554 30040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:09.489912 30040 hybrid_clock.cc:648] HybridClock initialized: now 1790563929489822 us; error 59 us; skew 500 ppm
I20260928 02:52:09.490276 30040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:09 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:09.494297 30040 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:09.496003 30040 webserver.cc:460] Webserver started at http://127.27.229.195:42933/ using document root <none> and password file <none>
I20260928 02:52:09.496796 30040 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:09.496942 30040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:09.497298 30040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:09.500597 30040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "51fa8f2c77324a8d8b49117611d112c8"
format_stamp: "Formatted at 2026-09-28 02:52:09 on dist-test-slave-s1kg"
server_key: "e5cf2a31e6c6f33d6e9a4ebc6525d23a"
server_key_iv: "d7c57c66bc0ffd8a054c58ab20edb2ee"
server_key_version: "encryptionkey@0"
I20260928 02:52:09.501482 30040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "51fa8f2c77324a8d8b49117611d112c8"
format_stamp: "Formatted at 2026-09-28 02:52:09 on dist-test-slave-s1kg"
server_key: "e5cf2a31e6c6f33d6e9a4ebc6525d23a"
server_key_iv: "d7c57c66bc0ffd8a054c58ab20edb2ee"
server_key_version: "encryptionkey@0"
I20260928 02:52:09.507413 30040 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 02:52:09.511807 30055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:09.513481 30040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:52:09.513679 30040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "51fa8f2c77324a8d8b49117611d112c8"
format_stamp: "Formatted at 2026-09-28 02:52:09 on dist-test-slave-s1kg"
server_key: "e5cf2a31e6c6f33d6e9a4ebc6525d23a"
server_key_iv: "d7c57c66bc0ffd8a054c58ab20edb2ee"
server_key_version: "encryptionkey@0"
I20260928 02:52:09.513922 30040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:09.542639 29897 heartbeater.cc:503] Master 127.27.229.254:42665 was elected leader, sending a full tablet report...
I20260928 02:52:09.566246 30040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:09.571705 30040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:09.571985 30040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:09.574971 30040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:09.575079 30040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:09.575259 30040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:09.575325 30040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:52:09 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:09.604887 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.575914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:59439 (local address 127.27.229.254:42665)
0928 02:52:09.576417 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.576424 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.577898 (+ 1474us) server_negotiation.cc:407] Connection header received
0928 02:52:09.580825 (+ 2927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.580831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.580951 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.581094 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:09.583997 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.585078 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.589282 (+ 4204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.589809 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.589987 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.596208 (+ 6221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:09.596214 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:09.596218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:09.596256 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:09.600841 (+ 4585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.601974 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.601981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.601984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.602175 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.602952 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.602957 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.602961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.603281 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:09.603449 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.604244 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.604472 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":242,"threads_started":1}
I20260928 02:52:09.605960 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.576681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.195:59439)
0928 02:52:09.577688 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.577742 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.580441 (+ 2699us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.581424 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.581441 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.582546 (+ 1105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:09.583765 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.583793 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.585357 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.585367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.589029 (+ 3662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.589058 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.590216 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.590225 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.590420 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.591413 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:09.591456 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:09.595790 (+ 4334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:09.601077 (+ 5287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.601093 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.601120 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.601797 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.602310 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.602318 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.602325 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.602757 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.603442 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:09.603451 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:09.603848 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.604132 (+ 284us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.604712 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":443,"threads_started":1}
I20260928 02:52:09.630151 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.613826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:33899 (local address 127.27.229.254:42665)
0928 02:52:09.614127 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.614137 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.614818 (+ 681us) server_negotiation.cc:407] Connection header received
0928 02:52:09.615079 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.615083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.615151 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.615242 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:09.616796 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.617595 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.620180 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.620806 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.620955 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.623602 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:09.623609 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:09.623614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:09.623660 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:09.627457 (+ 3797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.628179 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.628185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.628189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.628327 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.628709 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.628714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.628717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.628983 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:09.629163 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.629559 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.629773 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:52:09.630254 30141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.613847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.27.229.195:33899)
0928 02:52:09.614662 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.614688 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.614921 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.615316 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.615324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.616128 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:09.616608 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.616625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.617740 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.617748 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.619985 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.620006 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.621077 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.621087 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.621267 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.621997 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:09.622019 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:09.623326 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:09.627608 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.627613 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.627618 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.628039 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.628443 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:09.628448 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:09.628452 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:09.628590 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:09.629112 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:09.629117 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:09.629309 (+ 192us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.629449 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.629789 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":395,"threads_started":1}
I20260928 02:52:09.634577 30175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:34137 every 8 connection(s)
I20260928 02:52:09.634570 30040 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:34137
I20260928 02:52:09.636387 30040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:09.638797 30176 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42665
I20260928 02:52:09.639113 30176 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:09.639981 30176 heartbeater.cc:511] Master 127.27.229.254:42665 requested a full tablet report, sending...
I20260928 02:52:09.641541 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30040
I20260928 02:52:09.641629 29690 ts_manager.cc:194] Registered new tserver with Master: 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195:34137)
I20260928 02:52:09.641770 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:09.642323 28567 external_mini_cluster.cc:1428] Setting key cfe5001bccecd91744b064964f0ff810
I20260928 02:52:09.642798 29690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:59439
I20260928 02:52:09.645460 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:52:09 dist-test-slave-s1kg krb5kdc[29645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:52:09 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:09.693439 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.679143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57692 (local address 127.27.229.254:42665)
0928 02:52:09.679402 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.679409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.679638 (+ 229us) server_negotiation.cc:407] Connection header received
0928 02:52:09.679871 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.679875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.679940 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.680049 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:09.681407 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.682246 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.683670 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.684079 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.684215 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.688225 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:09.688230 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:09.688234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:09.688269 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:09.690718 (+ 2449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.691445 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.691452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.691462 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.691584 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:09.691989 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:09.691993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:09.691995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:09.692275 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:09.692543 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.692858 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.693074 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 02:52:09.698010 29690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:09.704803 29690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:09.748678 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.718975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51376 (local address 127.27.229.193:41347)
0928 02:52:09.728275 (+ 9300us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.728289 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.740556 (+ 12267us) server_negotiation.cc:407] Connection header received
0928 02:52:09.740727 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.740734 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.740967 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.741177 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:09.742125 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.743466 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.746109 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.747405 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.747583 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.747779 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.748148 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.748339 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9061,"thread_start_us":8882,"threads_started":1}
I20260928 02:52:09.749779 30189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.718289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41347 (local address 127.0.0.1:51376)
0928 02:52:09.740429 (+ 22140us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.740448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.740580 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.741390 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.741394 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.741445 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:09.741872 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.741884 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.743628 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.743632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.745973 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.745983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.747624 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.747631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.747851 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.747905 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.748992 (+ 1087us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.749206 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22017,"mutex_wait_us":80,"spinlock_wait_cycles":1280,"thread_start_us":16819,"threads_started":1}
I20260928 02:52:09.749806 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.718481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51776 (local address 127.27.229.194:46761)
0928 02:52:09.739501 (+ 21020us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.739514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.740396 (+ 882us) server_negotiation.cc:407] Connection header received
0928 02:52:09.740524 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.740535 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.740871 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.741104 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:09.742335 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.744002 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.747006 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.748401 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.748565 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.748733 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.749345 (+ 612us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.749500 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20768,"thread_start_us":20602,"threads_started":1}
I20260928 02:52:09.749864 30190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.717960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:46761 (local address 127.0.0.1:51776)
0928 02:52:09.740043 (+ 22083us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.740069 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.740238 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.741369 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.741378 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.741410 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:09.742122 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.742144 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.744188 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.744194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.746817 (+ 2623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.746853 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.749040 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.749046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.749164 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.749187 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.749304 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.749439 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21925,"mutex_wait_us":4,"thread_start_us":21784,"threads_started":1}
I20260928 02:52:09.750413 30192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.717772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34137 (local address 127.0.0.1:56378)
0928 02:52:09.734959 (+ 17187us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.735041 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.736338 (+ 1297us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.742002 (+ 5664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.742010 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.742042 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:09.742442 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.742474 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.744616 (+ 2142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.744630 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.747081 (+ 2451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.747091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.749340 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.749349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.749607 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.749653 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.749773 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.749933 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16735,"thread_start_us":22379,"threads_started":1}
I20260928 02:52:09.750608 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.717982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56378 (local address 127.27.229.195:34137)
0928 02:52:09.741291 (+ 23309us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.741301 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.741335 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:52:09.741427 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.741434 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.741774 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.741990 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:09.742686 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.744441 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.747245 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.748800 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.749000 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.749154 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.750038 (+ 884us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.750231 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23094,"thread_start_us":23421,"threads_started":1}
I20260928 02:52:09.754233 29827 tablet_service.cc:1467] Processing CreateTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 (DEFAULT_TABLE table=test-table [id=1f565dcb09864134ad2937083f21548f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:09.754256 30106 tablet_service.cc:1467] Processing CreateTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 (DEFAULT_TABLE table=test-table [id=1f565dcb09864134ad2937083f21548f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:09.755935 29827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b72538c57324495a5cc9e7e6d4e01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:09.755956 30106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b72538c57324495a5cc9e7e6d4e01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:09.757268 29966 tablet_service.cc:1467] Processing CreateTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 (DEFAULT_TABLE table=test-table [id=1f565dcb09864134ad2937083f21548f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:09.758535 29966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b72538c57324495a5cc9e7e6d4e01c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:09.769387 30195 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Bootstrap starting.
I20260928 02:52:09.771931 30196 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Bootstrap starting.
I20260928 02:52:09.772701 30197 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Bootstrap starting.
I20260928 02:52:09.773875 30195 tablet_bootstrap.cc:654] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:09.775238 30195 log.cc:824] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:09.776275 30197 tablet_bootstrap.cc:654] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:09.777542 30197 log.cc:824] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:09.777878 30196 tablet_bootstrap.cc:654] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:09.778502 30195 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: No bootstrap required, opened a new log
I20260928 02:52:09.778810 30195 ts_tablet_manager.cc:1397] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:52:09.779156 30196 log.cc:824] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:09.781409 30197 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: No bootstrap required, opened a new log
I20260928 02:52:09.781687 30197 ts_tablet_manager.cc:1397] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:52:09.785061 30196 tablet_bootstrap.cc:492] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: No bootstrap required, opened a new log
I20260928 02:52:09.785346 30196 ts_tablet_manager.cc:1397] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20260928 02:52:09.789672 30195 raft_consensus.cc:348] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.790145 30195 raft_consensus.cc:374] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:09.790235 30195 raft_consensus.cc:729] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdbcbca5d33499fa20a0543ef93be5e, State: Initialized, Role: FOLLOWER
I20260928 02:52:09.791033 30195 consensus_queue.cc:260] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.792213 30036 heartbeater.cc:503] Master 127.27.229.254:42665 was elected leader, sending a full tablet report...
I20260928 02:52:09.792986 30196 raft_consensus.cc:348] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.792899 30197 raft_consensus.cc:348] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.793344 30197 raft_consensus.cc:374] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:09.793357 30196 raft_consensus.cc:374] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:09.793450 30196 raft_consensus.cc:729] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ecea8616dd4af886ac208edd8b9909, State: Initialized, Role: FOLLOWER
I20260928 02:52:09.793450 30197 raft_consensus.cc:729] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa8f2c77324a8d8b49117611d112c8, State: Initialized, Role: FOLLOWER
I20260928 02:52:09.794128 30197 consensus_queue.cc:260] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.794134 30196 consensus_queue.cc:260] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.794533 30195 ts_tablet_manager.cc:1428] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260928 02:52:09.795701 30176 heartbeater.cc:503] Master 127.27.229.254:42665 was elected leader, sending a full tablet report...
I20260928 02:52:09.797086 30197 ts_tablet_manager.cc:1428] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Time spent starting tablet: real 0.015s user 0.013s sys 0.002s
I20260928 02:52:09.798192 30196 ts_tablet_manager.cc:1428] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
W20260928 02:52:09.815001 30037 tablet.cc:2369] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:09.888516 30177 tablet.cc:2369] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:09.945657 30202 raft_consensus.cc:482] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:09.946002 30202 raft_consensus.cc:504] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.947943 30202 leader_election.cc:288] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761), 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347)
I20260928 02:52:09.954088 30193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.948392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:49131 (local address 127.27.229.194:46761)
0928 02:52:09.948613 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.948620 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.948884 (+ 264us) server_negotiation.cc:407] Connection header received
0928 02:52:09.949107 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.949119 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.949178 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.949278 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:09.949701 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.950398 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.952263 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.953129 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.953323 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.953405 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.953671 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.953805 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:09.954286 30065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.948400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:46761 (local address 127.27.229.195:49131)
0928 02:52:09.948778 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.948796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.948957 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.949295 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.949299 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.949324 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:09.949531 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.949541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.950537 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.950545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.952073 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.952085 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.953307 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.953314 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.953508 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.953533 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.953644 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.953791 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":1}
I20260928 02:52:09.954993 30194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.948427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:38039 (local address 127.27.229.193:41347)
0928 02:52:09.948634 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.948642 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.949335 (+ 693us) server_negotiation.cc:407] Connection header received
0928 02:52:09.949574 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.949578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.949635 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.949734 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:09.950236 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.951157 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.953065 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.953954 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.954146 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.954220 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.954582 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.954724 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:09.955657 30204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.948442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41347 (local address 127.27.229.195:38039)
0928 02:52:09.949222 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:09.949236 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:09.949398 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:09.949747 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:09.949751 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:09.949774 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:09.950069 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.950077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.951301 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.951308 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.952878 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:09.952890 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.954167 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.954177 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:09.954372 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.954411 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:52:09.955097 (+ 686us) client_negotiation.cc:240] Negotiation successful
0928 02:52:09.955310 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":470,"threads_started":1}
I20260928 02:52:09.955618 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b72538c57324495a5cc9e7e6d4e01c0" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdbcbca5d33499fa20a0543ef93be5e" is_pre_election: true
I20260928 02:52:09.956236 29990 raft_consensus.cc:2454] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 0.
I20260928 02:52:09.956810 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b72538c57324495a5cc9e7e6d4e01c0" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909" is_pre_election: true
I20260928 02:52:09.957077 30058 leader_election.cc:302] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51fa8f2c77324a8d8b49117611d112c8, abdbcbca5d33499fa20a0543ef93be5e; no voters:
I20260928 02:52:09.957360 29847 raft_consensus.cc:2454] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 0.
I20260928 02:52:09.957542 30202 raft_consensus.cc:2781] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:09.957706 30202 raft_consensus.cc:482] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:09.957859 30202 raft_consensus.cc:3037] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:09.960100 30202 raft_consensus.cc:504] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.960692 30202 leader_election.cc:288] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 election: Requested vote from peers abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761), 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347)
I20260928 02:52:09.961275 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b72538c57324495a5cc9e7e6d4e01c0" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdbcbca5d33499fa20a0543ef93be5e"
I20260928 02:52:09.961270 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b72538c57324495a5cc9e7e6d4e01c0" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909"
I20260928 02:52:09.961490 29847 raft_consensus.cc:3037] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:09.961490 29990 raft_consensus.cc:3037] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:09.964737 29990 raft_consensus.cc:2454] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 1.
I20260928 02:52:09.964737 29847 raft_consensus.cc:2454] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 1.
I20260928 02:52:09.965209 30056 leader_election.cc:302] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ecea8616dd4af886ac208edd8b9909, 51fa8f2c77324a8d8b49117611d112c8; no voters:
I20260928 02:52:09.965543 30202 raft_consensus.cc:2781] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:09.965991 30202 raft_consensus.cc:686] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 LEADER]: Becoming Leader. State: Replica: 51fa8f2c77324a8d8b49117611d112c8, State: Running, Role: LEADER
I20260928 02:52:09.966451 30202 consensus_queue.cc:237] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } }
I20260928 02:52:09.971530 29689 catalog_manager.cc:5495] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "51fa8f2c77324a8d8b49117611d112c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:09.997004 30191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:09.991153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56390 (local address 127.27.229.195:34137)
0928 02:52:09.991384 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:09.991390 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:09.991413 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:09.991710 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:09.991714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:09.991805 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:09.991961 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:09.992366 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.993132 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.994591 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:09.995121 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:09.995259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:09.995772 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:09.996308 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:09.996474 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:09.996617 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:52:09.996705 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:52:10.006053 29990 raft_consensus.cc:1261] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Refusing update from remote peer 51fa8f2c77324a8d8b49117611d112c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.006052 29847 raft_consensus.cc:1261] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Refusing update from remote peer 51fa8f2c77324a8d8b49117611d112c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.007129 30205 consensus_queue.cc:1035] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.007726 30202 consensus_queue.cc:1035] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.026119 30210 mvcc.cc:205] Tried to move back new op lower bound from 7334149857294282752 to 7334149857146277888. Current Snapshot: MvccSnapshot[applied={T|T < 7334149857294282752}]
I20260928 02:52:10.028497 30209 mvcc.cc:205] Tried to move back new op lower bound from 7334149857294282752 to 7334149857146277888. Current Snapshot: MvccSnapshot[applied={T|T < 7334149857294282752}]
I20260928 02:52:10.045288 29688 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:10.045634 29688 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:10.049901 29688 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Sending DeleteTablet for 3 replicas of tablet 9b72538c57324495a5cc9e7e6d4e01c0
I20260928 02:52:10.051102 29827 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:51376
I20260928 02:52:10.051116 30106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56378
I20260928 02:52:10.051118 29966 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b72538c57324495a5cc9e7e6d4e01c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:51776
I20260928 02:52:10.052032 30219 tablet_replica.cc:330] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: stopping tablet replica
I20260928 02:52:10.052608 30221 tablet_replica.cc:330] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: stopping tablet replica
I20260928 02:52:10.053128 30219 raft_consensus.cc:2229] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:10.053625 30221 raft_consensus.cc:2229] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:10.053730 29688 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:10.054510 29688 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:10.054787 30221 raft_consensus.cc:2258] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.055008 30219 raft_consensus.cc:2258] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.056442 30220 tablet_replica.cc:330] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: stopping tablet replica
I20260928 02:52:10.057389 30220 raft_consensus.cc:2229] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:10.058076 30221 ts_tablet_manager.cc:1905] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:10.058061 30220 raft_consensus.cc:2258] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.059505 30219 ts_tablet_manager.cc:1905] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:10.062597 29966 tablet_service.cc:1467] Processing CreateTablet for tablet fae27bafd00f4a3fadb675d523a62298 (DEFAULT_TABLE table=test-table [id=42057a0d3a90448cad95508a86cf0c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:10.063066 29966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae27bafd00f4a3fadb675d523a62298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.066617 30219 ts_tablet_manager.cc:1918] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:10.066797 30219 log.cc:1196] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/9b72538c57324495a5cc9e7e6d4e01c0
I20260928 02:52:10.067373 30219 ts_tablet_manager.cc:1939] T 9b72538c57324495a5cc9e7e6d4e01c0 P 38ecea8616dd4af886ac208edd8b9909: Deleting consensus metadata
I20260928 02:52:10.068145 30106 tablet_service.cc:1467] Processing CreateTablet for tablet fae27bafd00f4a3fadb675d523a62298 (DEFAULT_TABLE table=test-table [id=42057a0d3a90448cad95508a86cf0c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:10.068653 30106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae27bafd00f4a3fadb675d523a62298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.070344 30220 ts_tablet_manager.cc:1905] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:10.067377 29827 tablet_service.cc:1467] Processing CreateTablet for tablet fae27bafd00f4a3fadb675d523a62298 (DEFAULT_TABLE table=test-table [id=42057a0d3a90448cad95508a86cf0c69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:10.075220 29674 catalog_manager.cc:4841] TS 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347): tablet 9b72538c57324495a5cc9e7e6d4e01c0 (table test-table [id=1f565dcb09864134ad2937083f21548f]) successfully deleted
I20260928 02:52:10.075552 29827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae27bafd00f4a3fadb675d523a62298. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.080721 30195 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Bootstrap starting.
I20260928 02:52:10.081938 30221 ts_tablet_manager.cc:1918] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:10.082094 30221 log.cc:1196] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/9b72538c57324495a5cc9e7e6d4e01c0
I20260928 02:52:10.082533 30195 tablet_bootstrap.cc:654] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.082669 30221 ts_tablet_manager.cc:1939] T 9b72538c57324495a5cc9e7e6d4e01c0 P abdbcbca5d33499fa20a0543ef93be5e: Deleting consensus metadata
I20260928 02:52:10.084231 30196 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Bootstrap starting.
I20260928 02:52:10.084453 29676 catalog_manager.cc:4841] TS abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761): tablet 9b72538c57324495a5cc9e7e6d4e01c0 (table test-table [id=1f565dcb09864134ad2937083f21548f]) successfully deleted
I20260928 02:52:10.086195 30196 tablet_bootstrap.cc:654] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.087172 30197 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Bootstrap starting.
I20260928 02:52:10.088716 30197 tablet_bootstrap.cc:654] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.092258 30220 ts_tablet_manager.cc:1918] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:10.092497 30220 log.cc:1196] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/9b72538c57324495a5cc9e7e6d4e01c0
I20260928 02:52:10.093173 30220 ts_tablet_manager.cc:1939] T 9b72538c57324495a5cc9e7e6d4e01c0 P 51fa8f2c77324a8d8b49117611d112c8: Deleting consensus metadata
I20260928 02:52:10.100236 29677 catalog_manager.cc:4841] TS 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195:34137): tablet 9b72538c57324495a5cc9e7e6d4e01c0 (table test-table [id=1f565dcb09864134ad2937083f21548f]) successfully deleted
I20260928 02:52:10.100992 30195 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: No bootstrap required, opened a new log
I20260928 02:52:10.101184 30196 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: No bootstrap required, opened a new log
I20260928 02:52:10.101212 30197 tablet_bootstrap.cc:492] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: No bootstrap required, opened a new log
I20260928 02:52:10.101219 30195 ts_tablet_manager.cc:1397] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.004s
I20260928 02:52:10.101331 30196 ts_tablet_manager.cc:1397] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20260928 02:52:10.101367 30197 ts_tablet_manager.cc:1397] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.012s
I20260928 02:52:10.102150 30196 raft_consensus.cc:348] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.102150 30195 raft_consensus.cc:348] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.102214 30197 raft_consensus.cc:348] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.102380 30196 raft_consensus.cc:374] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.102438 30197 raft_consensus.cc:374] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.102485 30196 raft_consensus.cc:729] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ecea8616dd4af886ac208edd8b9909, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.102561 30197 raft_consensus.cc:729] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa8f2c77324a8d8b49117611d112c8, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.102674 30195 raft_consensus.cc:374] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.102814 30195 raft_consensus.cc:729] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdbcbca5d33499fa20a0543ef93be5e, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.102803 30196 consensus_queue.cc:260] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.102989 30197 consensus_queue.cc:260] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.104127 30196 ts_tablet_manager.cc:1428] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:10.104485 30195 consensus_queue.cc:260] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.105655 30195 ts_tablet_manager.cc:1428] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 02:52:10.105906 30197 ts_tablet_manager.cc:1428] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:52:10.279294 30216 raft_consensus.cc:482] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:10.279610 30216 raft_consensus.cc:504] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.280606 30216 leader_election.cc:288] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347), abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761)
I20260928 02:52:10.281111 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae27bafd00f4a3fadb675d523a62298" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909" is_pre_election: true
I20260928 02:52:10.281240 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae27bafd00f4a3fadb675d523a62298" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdbcbca5d33499fa20a0543ef93be5e" is_pre_election: true
I20260928 02:52:10.281381 29847 raft_consensus.cc:2454] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 0.
I20260928 02:52:10.281512 29990 raft_consensus.cc:2454] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 0.
I20260928 02:52:10.281840 30056 leader_election.cc:302] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ecea8616dd4af886ac208edd8b9909, 51fa8f2c77324a8d8b49117611d112c8; no voters:
I20260928 02:52:10.282121 30216 raft_consensus.cc:2781] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:10.282245 30216 raft_consensus.cc:482] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:10.282368 30216 raft_consensus.cc:3037] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:52:10.284725 29898 tablet.cc:2369] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:10.284818 30216 raft_consensus.cc:504] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.285775 30216 leader_election.cc:288] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 election: Requested vote from peers 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347), abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761)
I20260928 02:52:10.285900 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae27bafd00f4a3fadb675d523a62298" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909"
I20260928 02:52:10.285929 29990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae27bafd00f4a3fadb675d523a62298" candidate_uuid: "51fa8f2c77324a8d8b49117611d112c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "abdbcbca5d33499fa20a0543ef93be5e"
I20260928 02:52:10.286072 29847 raft_consensus.cc:3037] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:10.286150 29990 raft_consensus.cc:3037] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:10.288125 29847 raft_consensus.cc:2454] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 1.
I20260928 02:52:10.288625 30056 leader_election.cc:302] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ecea8616dd4af886ac208edd8b9909, 51fa8f2c77324a8d8b49117611d112c8; no voters:
I20260928 02:52:10.288914 30216 raft_consensus.cc:2781] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:10.289085 30216 raft_consensus.cc:686] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 LEADER]: Becoming Leader. State: Replica: 51fa8f2c77324a8d8b49117611d112c8, State: Running, Role: LEADER
I20260928 02:52:10.289095 29990 raft_consensus.cc:2454] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51fa8f2c77324a8d8b49117611d112c8 in term 1.
I20260928 02:52:10.289377 30216 consensus_queue.cc:237] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.292189 29690 catalog_manager.cc:5495] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "51fa8f2c77324a8d8b49117611d112c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } health_report { overall_health: HEALTHY } } }
Sep 28 02:52:10 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563927, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:10.373641 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.357546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.27.229.254:42665)
0928 02:52:10.358638 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.358642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.358663 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:10.358707 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.358710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.358765 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.358918 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.360389 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.361179 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.363103 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.363656 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.363755 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.367950 (+ 4195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.367955 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.367958 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.367989 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.370795 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.371620 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.371625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.371629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.371756 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.372132 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.372140 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.372143 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.372393 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.372540 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.372925 (+ 385us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.373173 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":997,"spinlock_wait_cycles":33536,"thread_start_us":858,"threads_started":1}
I20260928 02:52:10.373720 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.357530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42665 (local address 127.0.0.1:57700)
0928 02:52:10.358275 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.358307 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.358474 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.358962 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.358970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.359929 (+ 959us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:10.360194 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.360206 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.361330 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.361335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.362943 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.362958 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.363819 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.363828 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.364017 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.364817 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:10.364848 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:10.367711 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:10.370988 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:10.371002 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:10.371010 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:10.371490 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:10.371858 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:10.371865 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:10.371869 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:10.372030 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:10.372513 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:10.372522 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:10.372687 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.372853 (+ 166us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.373308 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":170,"thread_start_us":382,"threads_started":1}
I20260928 02:52:10.379472 29690 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:57700:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:52:10.380735 29690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:10.388016 29827 tablet_service.cc:1467] Processing CreateTablet for tablet cf34e8f93c874924a1e19d3bf47f23b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d10e6c76534bc4a59935d249bfa995]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:10.388183 30106 tablet_service.cc:1467] Processing CreateTablet for tablet cf34e8f93c874924a1e19d3bf47f23b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d10e6c76534bc4a59935d249bfa995]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:10.388645 29827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34e8f93c874924a1e19d3bf47f23b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.388756 30106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34e8f93c874924a1e19d3bf47f23b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.394146 29966 tablet_service.cc:1467] Processing CreateTablet for tablet cf34e8f93c874924a1e19d3bf47f23b3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=77d10e6c76534bc4a59935d249bfa995]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:10.394423 30197 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: Bootstrap starting.
I20260928 02:52:10.394726 29966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf34e8f93c874924a1e19d3bf47f23b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:10.396024 30196 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: Bootstrap starting.
I20260928 02:52:10.396246 30197 tablet_bootstrap.cc:654] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.397716 30196 tablet_bootstrap.cc:654] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.405722 30195 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: Bootstrap starting.
I20260928 02:52:10.406931 30196 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: No bootstrap required, opened a new log
I20260928 02:52:10.407119 30196 ts_tablet_manager.cc:1397] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260928 02:52:10.407330 30195 tablet_bootstrap.cc:654] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:10.407889 30196 raft_consensus.cc:348] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.408150 30196 raft_consensus.cc:374] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.408238 30196 raft_consensus.cc:729] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ecea8616dd4af886ac208edd8b9909, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.408515 30196 consensus_queue.cc:260] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.409174 30196 ts_tablet_manager.cc:1428] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:52:10.409044 30203 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.409457 30203 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.410629 30197 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: No bootstrap required, opened a new log
I20260928 02:52:10.410856 30197 ts_tablet_manager.cc:1397] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20260928 02:52:10.411721 30197 raft_consensus.cc:348] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.411968 30197 raft_consensus.cc:374] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.412050 30197 raft_consensus.cc:729] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51fa8f2c77324a8d8b49117611d112c8, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.412334 30197 consensus_queue.cc:260] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.412959 30197 ts_tablet_manager.cc:1428] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260928 02:52:10.412812 30216 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.413233 30216 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.416559 30195 tablet_bootstrap.cc:492] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: No bootstrap required, opened a new log
I20260928 02:52:10.416746 30195 ts_tablet_manager.cc:1397] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.006s
I20260928 02:52:10.417575 30195 raft_consensus.cc:348] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.417825 30195 raft_consensus.cc:374] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:10.417913 30195 raft_consensus.cc:729] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abdbcbca5d33499fa20a0543ef93be5e, State: Initialized, Role: FOLLOWER
I20260928 02:52:10.418193 30195 consensus_queue.cc:260] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.418749 30201 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.419258 30201 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.419440 30195 ts_tablet_manager.cc:1428] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:52:10.529028 30201 raft_consensus.cc:482] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:10.529264 30201 raft_consensus.cc:504] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.531172 30201 leader_election.cc:288] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347), 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195:34137)
I20260928 02:52:10.555449 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.531687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:55681 (local address 127.27.229.193:41347)
0928 02:52:10.549543 (+ 17856us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.549550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.549571 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:10.549652 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.549658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.549728 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.549841 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:10.550361 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.551591 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.553423 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.554414 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.554583 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.554664 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.554956 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.555092 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17716,"thread_start_us":18229,"threads_started":1}
I20260928 02:52:10.555540 30236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.531541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41347 (local address 127.27.229.194:55681)
0928 02:52:10.546416 (+ 14875us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.546463 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.546655 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.549893 (+ 3238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.549901 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.549928 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:10.550204 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.550217 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.551754 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.551760 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.553278 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.553290 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.554574 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.554580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.554773 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.554797 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.554933 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.555119 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14710,"mutex_wait_us":23,"thread_start_us":17901,"threads_started":1}
I20260928 02:52:10.556377 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34e8f93c874924a1e19d3bf47f23b3" candidate_uuid: "abdbcbca5d33499fa20a0543ef93be5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909" is_pre_election: true
I20260928 02:52:10.556675 29847 raft_consensus.cc:2454] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abdbcbca5d33499fa20a0543ef93be5e in term 0.
I20260928 02:52:10.557279 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.531613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34137 (local address 127.27.229.194:48455)
0928 02:52:10.549696 (+ 18083us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.549713 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.549842 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.550651 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.550656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.550682 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:10.551036 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.551046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.552118 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.552124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.554602 (+ 2478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.554618 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.556420 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.556425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.556595 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.556627 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.556737 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.556866 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17961,"mutex_wait_us":56,"thread_start_us":14470,"threads_started":1}
I20260928 02:52:10.557255 29917 leader_election.cc:302] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ecea8616dd4af886ac208edd8b9909, abdbcbca5d33499fa20a0543ef93be5e; no voters:
I20260928 02:52:10.557713 30201 raft_consensus.cc:2781] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:10.557315 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.531972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:48455 (local address 127.27.229.195:34137)
0928 02:52:10.550369 (+ 18397us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.550375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.550395 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:52:10.550455 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.550458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.550512 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.550622 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:10.551185 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.551969 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.554815 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.556211 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.556371 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.556445 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.556845 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.557002 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18272,"thread_start_us":18177,"threads_started":1}
I20260928 02:52:10.557828 30201 raft_consensus.cc:482] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:10.557922 30201 raft_consensus.cc:3037] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:10.558599 30126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34e8f93c874924a1e19d3bf47f23b3" candidate_uuid: "abdbcbca5d33499fa20a0543ef93be5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fa8f2c77324a8d8b49117611d112c8" is_pre_election: true
I20260928 02:52:10.559015 30126 raft_consensus.cc:2454] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate abdbcbca5d33499fa20a0543ef93be5e in term 0.
I20260928 02:52:10.560370 30201 raft_consensus.cc:504] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.560976 30201 leader_election.cc:288] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [CANDIDATE]: Term 1 election: Requested vote from peers 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347), 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195:34137)
I20260928 02:52:10.561368 29847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34e8f93c874924a1e19d3bf47f23b3" candidate_uuid: "abdbcbca5d33499fa20a0543ef93be5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ecea8616dd4af886ac208edd8b9909"
I20260928 02:52:10.561467 30126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf34e8f93c874924a1e19d3bf47f23b3" candidate_uuid: "abdbcbca5d33499fa20a0543ef93be5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "51fa8f2c77324a8d8b49117611d112c8"
I20260928 02:52:10.561589 29847 raft_consensus.cc:3037] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:10.561671 30126 raft_consensus.cc:3037] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:10.563939 29847 raft_consensus.cc:2454] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abdbcbca5d33499fa20a0543ef93be5e in term 1.
I20260928 02:52:10.564348 29917 leader_election.cc:302] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ecea8616dd4af886ac208edd8b9909, abdbcbca5d33499fa20a0543ef93be5e; no voters:
I20260928 02:52:10.564625 30201 raft_consensus.cc:2781] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:10.565011 30126 raft_consensus.cc:2454] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate abdbcbca5d33499fa20a0543ef93be5e in term 1.
I20260928 02:52:10.565086 30201 raft_consensus.cc:686] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [term 1 LEADER]: Becoming Leader. State: Replica: abdbcbca5d33499fa20a0543ef93be5e, State: Running, Role: LEADER
I20260928 02:52:10.565511 30201 consensus_queue.cc:237] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } }
I20260928 02:52:10.567479 30240 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: TxnStatusTablet state changed. Reason: New leader abdbcbca5d33499fa20a0543ef93be5e. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.567929 30240 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:10.568720 30242 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:52:10.569000 30242 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:52:10.569581 29690 catalog_manager.cc:5495] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e reported cstate change: term changed from 0 to 1, leader changed from <none> to abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194). New cstate: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:10.624297 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.616909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51782 (local address 127.27.229.194:46761)
0928 02:52:10.617693 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.617700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.617723 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:10.617824 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.617831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.617945 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.618140 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:10.618632 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.619972 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.621257 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.622000 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.622153 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.622705 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.623595 (+ 890us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.623858 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.623919 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.624014 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":486,"threads_started":1}
I20260928 02:52:10.624416 30232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.616961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:46761 (local address 127.0.0.1:51782)
0928 02:52:10.617309 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.617337 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.617494 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.618150 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.618154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.618164 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:10.618383 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.618394 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.620116 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.620120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.621112 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.621123 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.622173 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.622177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.622354 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.622409 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.623749 (+ 1340us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.623778 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.623862 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.624041 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 02:52:10.629038 29847 raft_consensus.cc:1261] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Refusing update from remote peer abdbcbca5d33499fa20a0543ef93be5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.629251 30126 raft_consensus.cc:1261] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Refusing update from remote peer abdbcbca5d33499fa20a0543ef93be5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.629905 30240 consensus_queue.cc:1035] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.630246 30201 consensus_queue.cc:1035] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.639925 30203 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: TxnStatusTablet state changed. Reason: New leader abdbcbca5d33499fa20a0543ef93be5e. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.640256 30203 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.648686 30201 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.649291 30201 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:10.658370 30216 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: TxnStatusTablet state changed. Reason: New leader abdbcbca5d33499fa20a0543ef93be5e. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.658746 30216 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.661291 30240 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.661581 30240 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P abdbcbca5d33499fa20a0543ef93be5e: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:10.666272 30203 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.666527 30203 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 38ecea8616dd4af886ac208edd8b9909: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.683274 30216 tablet_replica.cc:439] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "abdbcbca5d33499fa20a0543ef93be5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 } } peers { permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 } } peers { permanent_uuid: "51fa8f2c77324a8d8b49117611d112c8" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34137 } } }
I20260928 02:52:10.683566 30216 tablet_replica.cc:442] T cf34e8f93c874924a1e19d3bf47f23b3 P 51fa8f2c77324a8d8b49117611d112c8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:10.699129 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.693548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:36269 (local address 127.27.229.194:46761)
0928 02:52:10.693762 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.693770 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.694263 (+ 493us) server_negotiation.cc:407] Connection header received
0928 02:52:10.694493 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.694498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.694576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.694696 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:10.695205 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.695955 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.697312 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.697768 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.697931 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.698364 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.698479 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.698578 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.698722 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.698800 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:52:10.699229 30256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.693526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:46761 (local address 127.27.229.195:36269)
0928 02:52:10.694147 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.694163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.694336 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.694713 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.694717 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.694729 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:10.695059 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.695069 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.696121 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.696126 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.697169 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.697180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.697979 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.697987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.698128 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.698182 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.698578 (+ 396us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.698592 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.698690 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.698825 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":340,"threads_started":1}
I20260928 02:52:10.709728 29990 raft_consensus.cc:1261] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Refusing update from remote peer 51fa8f2c77324a8d8b49117611d112c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.709802 29847 raft_consensus.cc:1261] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Refusing update from remote peer 51fa8f2c77324a8d8b49117611d112c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:10.710477 30205 consensus_queue.cc:1035] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "abdbcbca5d33499fa20a0543ef93be5e" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.710749 30216 consensus_queue.cc:1035] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ecea8616dd4af886ac208edd8b9909" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:10.721287 30248 mvcc.cc:205] Tried to move back new op lower bound from 7334149860181491712 to 7334149858467844096. Current Snapshot: MvccSnapshot[applied={T|T < 7334149860181491712}]
I20260928 02:52:10.754331 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.747194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37355 (local address 127.27.229.195:34137)
0928 02:52:10.747409 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.747416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.747852 (+ 436us) server_negotiation.cc:407] Connection header received
0928 02:52:10.748066 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.748071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.748156 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.748251 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:10.748820 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.749531 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.751386 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.751808 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.751946 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.752378 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.753686 (+ 1308us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.753795 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.753948 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.754032 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:52:10.754593 30262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.747087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34137 (local address 127.27.229.194:37355)
0928 02:52:10.747746 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:10.747763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:10.747930 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:10.748264 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:10.748269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:10.748280 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:10.748674 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.748686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.749673 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.749678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.751227 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:10.751243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.751998 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.752006 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:10.752137 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.752189 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:10.753795 (+ 1606us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:10.753819 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:52:10.753915 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:52:10.754100 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":284,"threads_started":1}
I20260928 02:52:10.801177 29690 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:10.801442 29690 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:10.804414 29690 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eaa9d888458c481187c1a12f2555e79f: Sending DeleteTablet for 3 replicas of tablet fae27bafd00f4a3fadb675d523a62298
I20260928 02:52:10.805289 29827 tablet_service.cc:1514] Processing DeleteTablet for tablet fae27bafd00f4a3fadb675d523a62298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:51376
I20260928 02:52:10.805402 29966 tablet_service.cc:1514] Processing DeleteTablet for tablet fae27bafd00f4a3fadb675d523a62298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:51776
I20260928 02:52:10.805990 30264 tablet_replica.cc:330] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: stopping tablet replica
I20260928 02:52:10.805992 30106 tablet_service.cc:1514] Processing DeleteTablet for tablet fae27bafd00f4a3fadb675d523a62298 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:10 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56378
I20260928 02:52:10.806318 30264 raft_consensus.cc:2229] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:10.806695 30264 raft_consensus.cc:2258] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.808008 30264 ts_tablet_manager.cc:1905] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:10.817909 30264 ts_tablet_manager.cc:1918] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:10.818207 30264 log.cc:1196] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/fae27bafd00f4a3fadb675d523a62298
I20260928 02:52:10.818962 30264 ts_tablet_manager.cc:1939] T fae27bafd00f4a3fadb675d523a62298 P 38ecea8616dd4af886ac208edd8b9909: Deleting consensus metadata
I20260928 02:52:10.820546 29674 catalog_manager.cc:4841] TS 38ecea8616dd4af886ac208edd8b9909 (127.27.229.193:41347): tablet fae27bafd00f4a3fadb675d523a62298 (table test-table [id=42057a0d3a90448cad95508a86cf0c69]) successfully deleted
I20260928 02:52:10.820916 30266 tablet_replica.cc:330] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: stopping tablet replica
I20260928 02:52:10.821146 30266 raft_consensus.cc:2229] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:10.821511 30266 raft_consensus.cc:2258] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.822697 30266 ts_tablet_manager.cc:1905] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:10.830442 30266 ts_tablet_manager.cc:1918] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:10.830710 30266 log.cc:1196] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/fae27bafd00f4a3fadb675d523a62298
I20260928 02:52:10.831311 30266 ts_tablet_manager.cc:1939] T fae27bafd00f4a3fadb675d523a62298 P 51fa8f2c77324a8d8b49117611d112c8: Deleting consensus metadata
I20260928 02:52:10.831408 30265 tablet_replica.cc:330] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: stopping tablet replica
I20260928 02:52:10.831610 30265 raft_consensus.cc:2229] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:10.831908 30265 raft_consensus.cc:2258] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:10.832635 29677 catalog_manager.cc:4841] TS 51fa8f2c77324a8d8b49117611d112c8 (127.27.229.195:34137): tablet fae27bafd00f4a3fadb675d523a62298 (table test-table [id=42057a0d3a90448cad95508a86cf0c69]) successfully deleted
I20260928 02:52:10.833240 30265 ts_tablet_manager.cc:1905] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:52:10 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:10.840508 30265 ts_tablet_manager.cc:1918] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:10.840694 30265 log.cc:1196] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/fae27bafd00f4a3fadb675d523a62298
I20260928 02:52:10.841233 30265 ts_tablet_manager.cc:1939] T fae27bafd00f4a3fadb675d523a62298 P abdbcbca5d33499fa20a0543ef93be5e: Deleting consensus metadata
I20260928 02:52:10.842384 29676 catalog_manager.cc:4841] TS abdbcbca5d33499fa20a0543ef93be5e (127.27.229.194:46761): tablet fae27bafd00f4a3fadb675d523a62298 (table test-table [id=42057a0d3a90448cad95508a86cf0c69]) successfully deleted
I20260928 02:52:10.843552 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.819112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51390 (local address 127.27.229.193:41347)
0928 02:52:10.819336 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.819345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.822277 (+ 2932us) server_negotiation.cc:407] Connection header received
0928 02:52:10.822603 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.822614 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.822719 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.822809 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.824878 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.825930 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.827976 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.828608 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.828721 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.835366 (+ 6645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.835387 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.835411 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.835463 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.839822 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.840852 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.840865 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.840869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.841044 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.841574 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.841580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.841585 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.842106 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.842439 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.842887 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.843156 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 02:52:10.844497 29892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51390
I20260928 02:52:10.860769 30231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.847747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.27.229.254:42665)
0928 02:52:10.848029 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.848035 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.848209 (+ 174us) server_negotiation.cc:407] Connection header received
0928 02:52:10.848417 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.848423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.848500 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.848613 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.849861 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.850575 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.851867 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.852292 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.852424 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.855118 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.855124 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.855131 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.855172 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.857949 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.858669 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.858676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.858680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.858817 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.859293 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.859298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.859302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.859607 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.859857 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.860195 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.860420 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
W20260928 02:52:10.861408 29690 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57708
I20260928 02:52:10.864084 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29761
I20260928 02:52:10.879050 30238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.865034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.27.229.193:41347)
0928 02:52:10.865225 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.865230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.867288 (+ 2058us) server_negotiation.cc:407] Connection header received
0928 02:52:10.867512 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.867516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.867612 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.867735 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.868918 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.869641 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.870917 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.871332 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.871474 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.873705 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.873710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.873713 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.873745 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.876490 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.877134 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.877140 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.877144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.877261 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.877641 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.877647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.877651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.877912 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.878179 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.878458 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.878688 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:52:10.879710 29892 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51400
W20260928 02:52:10.880246 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29761: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:10.880442 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29761
I20260928 02:52:10.906006 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29901
Sep 28 02:52:10 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:10.925706 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.907049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51790 (local address 127.27.229.194:46761)
0928 02:52:10.907234 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.907240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.907311 (+ 71us) server_negotiation.cc:407] Connection header received
0928 02:52:10.907551 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.907557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.907730 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.907851 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.909453 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.910661 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.912409 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.913111 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.913248 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.918082 (+ 4834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.918095 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.918118 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.918174 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.922309 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.923210 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.923222 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.923231 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.923413 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.923849 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.923856 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.923859 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.924327 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.924635 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.925058 (+ 423us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.925310 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 02:52:10.926512 30031 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51790
W20260928 02:52:10.927294 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29901: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:10.927464 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29901
I20260928 02:52:10.948920 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30040
Sep 28 02:52:10 dist-test-slave-s1kg krb5kdc[29645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:10.965649 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:10.949693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.27.229.195:34137)
0928 02:52:10.949897 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:10.949905 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:10.950010 (+ 105us) server_negotiation.cc:407] Connection header received
0928 02:52:10.950263 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:10.950269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:10.950364 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:10.950482 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:10.951936 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.952710 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.954088 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:10.954551 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:10.954705 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:10.959799 (+ 5094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:10.959815 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:10.959832 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:10.959869 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:10.962766 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.963558 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.963569 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.963573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.963731 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:10.964113 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:10.964118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:10.964121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:10.964484 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:10.964806 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:10.965083 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:52:10.965307 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:52:10.966357 30171 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56400
W20260928 02:52:10.967053 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30040: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:10.967240 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30040
I20260928 02:52:10.988364 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 29658
I20260928 02:52:10.989542 29730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:11.104104 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 29658
2026-09-28T02:52:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (7138 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:11 dist-test-slave-s1kg krb5kdc[30275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:11 dist-test-slave-s1kg krb5kdc[30275](info): set up 2 sockets
Sep 28 02:52:11 dist-test-slave-s1kg krb5kdc[30275](info): commencing operation
krb5kdc: starting...
W20260928 02:52:12.203218 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:12 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
W20260928 02:52:13.373139 28567 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.229.254: real 1.014s user 0.004s sys 0.004s
I20260928 02:52:13.374191 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34959
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:13.704176 30288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:13.704478 30288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:13.704644 30288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:13.711910 30288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:13.712019 30288 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:52:13.712095 30288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:13.712196 30288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:13.712292 30288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:13.712386 30288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:13.722281 30288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34959
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:13.723675 30288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:13.725076 30288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:13.737870 30294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:13.737951 30293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:13.737951 30296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:13.739357 30288 server_base.cc:1034] running on GCE node
I20260928 02:52:13.740162 30288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:13.741664 30288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:13.742964 30288 hybrid_clock.cc:648] HybridClock initialized: now 1790563933742893 us; error 34 us; skew 500 ppm
I20260928 02:52:13.743330 30288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:13 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:13.751106 30288 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:13.752779 30288 webserver.cc:460] Webserver started at http://127.27.229.254:36349/ using document root <none> and password file <none>
I20260928 02:52:13.753561 30288 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:13.753698 30288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:13.754030 30288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:13.757217 30288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "0d8739cf2ff040a5afe2deaabe272395"
format_stamp: "Formatted at 2026-09-28 02:52:13 on dist-test-slave-s1kg"
server_key: "74fc765b64395dd000aed242b53c683c"
server_key_iv: "090489f74587988aed6599d2d235588c"
server_key_version: "encryptionkey@0"
I20260928 02:52:13.758041 30288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "0d8739cf2ff040a5afe2deaabe272395"
format_stamp: "Formatted at 2026-09-28 02:52:13 on dist-test-slave-s1kg"
server_key: "74fc765b64395dd000aed242b53c683c"
server_key_iv: "090489f74587988aed6599d2d235588c"
server_key_version: "encryptionkey@0"
I20260928 02:52:13.763546 30288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:52:13.767164 30303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:13.768522 30288 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:13.768708 30288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "0d8739cf2ff040a5afe2deaabe272395"
format_stamp: "Formatted at 2026-09-28 02:52:13 on dist-test-slave-s1kg"
server_key: "74fc765b64395dd000aed242b53c683c"
server_key_iv: "090489f74587988aed6599d2d235588c"
server_key_version: "encryptionkey@0"
I20260928 02:52:13.768939 30288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:13.791064 30288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:13.795724 30288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:13.796005 30288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:13.822434 30288 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34959
I20260928 02:52:13.822441 30364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34959 every 8 connection(s)
I20260928 02:52:13.823961 30288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:13.828214 30365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:13.834307 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30288
I20260928 02:52:13.834570 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:13.835394 28567 external_mini_cluster.cc:1428] Setting key 5ed65c714e1377fa2a84f8689f164216
I20260928 02:52:13.840728 30365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Bootstrap starting.
I20260928 02:52:13.845086 30365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:13.846415 30365 log.cc:824] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:13.850077 30365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: No bootstrap required, opened a new log
Sep 28 02:52:13 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:13.860690 30365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } }
I20260928 02:52:13.861228 30365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:13.861382 30365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d8739cf2ff040a5afe2deaabe272395, State: Initialized, Role: FOLLOWER
I20260928 02:52:13.862087 30365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } }
I20260928 02:52:13.862350 30365 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:13.862525 30365 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:13.862728 30365 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:13.863590 30368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:13.837709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.27.229.254:34959)
0928 02:52:13.838709 (+ 1000us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:13.838723 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:13.838768 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:52:13.840686 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:13.840716 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:13.841219 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:13.841595 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:13.843862 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:13.845453 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:13.847463 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:13.848411 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:13.848561 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:13.853668 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:13.853680 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:13.853704 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:13.853771 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:13.858228 (+ 4457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:13.859364 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:13.859374 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:13.859387 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:13.859630 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:13.860028 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:13.860032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:13.860042 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:13.860777 (+ 735us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:13.861088 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:13.861576 (+ 488us) server_negotiation.cc:299] Negotiation successful
0928 02:52:13.862184 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":514,"threads_started":1}
I20260928 02:52:13.865404 30365 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } }
I20260928 02:52:13.865979 30365 leader_election.cc:302] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d8739cf2ff040a5afe2deaabe272395; no voters:
I20260928 02:52:13.866703 30365 leader_election.cc:288] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:13.866858 30370 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:13.867609 30370 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [term 1 LEADER]: Becoming Leader. State: Replica: 0d8739cf2ff040a5afe2deaabe272395, State: Running, Role: LEADER
I20260928 02:52:13.868237 30370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } }
I20260928 02:52:13.868916 30365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:13.871441 30371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d8739cf2ff040a5afe2deaabe272395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } } }
I20260928 02:52:13.871408 30372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d8739cf2ff040a5afe2deaabe272395. Latest consensus state: current_term: 1 leader_uuid: "0d8739cf2ff040a5afe2deaabe272395" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d8739cf2ff040a5afe2deaabe272395" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34959 } } }
I20260928 02:52:13.871918 30372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:13.871929 30371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:13.873320 30378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:13.879233 30378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:13.889254 30378 catalog_manager.cc:1347] Generated new cluster ID: 2eee33f21fb84a7dbb1b3c6d0dd5617c
I20260928 02:52:13.889396 30378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:13.916049 30378 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:13.918972 30378 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:13.936007 30378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Generated new TSK 0
I20260928 02:52:13.936956 30378 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:13.994119 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34959
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:14.329561 30391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:14.329856 30391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:14.330035 30391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:14.337352 30391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:14.337471 30391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:14.337638 30391 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:52:14.337702 30391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:14.337818 30391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:14.347826 30391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:14.349303 30391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:14.350592 30391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:14.364305 30397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:14.364351 30399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:14.364393 30396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:14.364945 30391 server_base.cc:1034] running on GCE node
I20260928 02:52:14.365741 30391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:14.366739 30391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:14.367961 30391 hybrid_clock.cc:648] HybridClock initialized: now 1790563934367915 us; error 40 us; skew 500 ppm
I20260928 02:52:14.368317 30391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:14 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:14.371841 30391 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:14.373489 30391 webserver.cc:460] Webserver started at http://127.27.229.193:39557/ using document root <none> and password file <none>
I20260928 02:52:14.374177 30391 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:14.374306 30391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:14.374634 30391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:14.377574 30391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "9067a2388744403d99630194b777d289"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "8471a7851b22119a42b56835fc5dd27d"
server_key_iv: "9be1334152f8436c07c6f5508777efa8"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.378352 30391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "9067a2388744403d99630194b777d289"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "8471a7851b22119a42b56835fc5dd27d"
server_key_iv: "9be1334152f8436c07c6f5508777efa8"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.383808 30391 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:52:14.387372 30406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:14.388833 30391 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:14.389019 30391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "9067a2388744403d99630194b777d289"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "8471a7851b22119a42b56835fc5dd27d"
server_key_iv: "9be1334152f8436c07c6f5508777efa8"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.389243 30391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:14.412849 30391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:14.417198 30391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:14.417523 30391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:14.420519 30391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:14.420665 30391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:14.420855 30391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:14.420948 30391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:52:14 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:14.446405 30416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.421771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.193:46185)
0928 02:52:14.422615 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:14.422654 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:14.424498 (+ 1844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:14.425308 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:14.425323 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:14.426173 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:14.427102 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.427121 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.428483 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.428493 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.431550 (+ 3057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.431577 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.432846 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.432854 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.433048 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.433813 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:14.433857 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:14.437291 (+ 3434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:14.440841 (+ 3550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.440856 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.440882 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.441748 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.442154 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.442162 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.442168 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.442482 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.443189 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:14.443201 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:14.443602 (+ 401us) client_negotiation.cc:769] Sending connection context
0928 02:52:14.443993 (+ 391us) client_negotiation.cc:240] Negotiation successful
0928 02:52:14.444641 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":353,"threads_started":1}
I20260928 02:52:14.447016 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.421037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:46185 (local address 127.27.229.254:34959)
0928 02:52:14.421553 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:14.421562 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:14.422807 (+ 1245us) server_negotiation.cc:407] Connection header received
0928 02:52:14.424816 (+ 2009us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:14.424821 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:14.424915 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:14.425033 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:14.427286 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.428261 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.431810 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.432355 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.432522 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.437548 (+ 5026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:14.437553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:14.437557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:14.437589 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:14.440678 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.441906 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.441912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.441915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.442040 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.442652 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.442657 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.442659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.443053 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:14.443205 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:14.446487 (+ 3282us) server_negotiation.cc:299] Negotiation successful
0928 02:52:14.446731 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":286,"threads_started":1}
I20260928 02:52:14.471051 30475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.453787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.193:34739)
0928 02:52:14.454558 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:14.454579 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:14.454761 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:14.455450 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:14.455457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:14.456376 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:14.456957 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.456974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.458245 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.458251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.460146 (+ 1895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.460158 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.461358 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.461363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.461615 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.462381 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:14.462411 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:14.463700 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:14.467352 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.467359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.467364 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.467987 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.468444 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.468449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.468453 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.468693 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.469375 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:14.469380 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:14.469673 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 02:52:14.469870 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 02:52:14.470436 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":397,"threads_started":1}
I20260928 02:52:14.471077 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.453810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:34739 (local address 127.27.229.254:34959)
0928 02:52:14.454058 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:14.454067 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:14.454690 (+ 623us) server_negotiation.cc:407] Connection header received
0928 02:52:14.454983 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:14.454992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:14.455092 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:14.455198 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:14.457147 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.458107 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.460403 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.460969 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.461126 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.463985 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:14.463994 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:14.463998 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:14.464051 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:14.467174 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.468136 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.468142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.468146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.468302 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.468829 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.468837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.468841 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.469183 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:14.469426 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:14.470463 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 02:52:14.470733 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:14.472931 30391 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37367
I20260928 02:52:14.472980 30526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37367 every 8 connection(s)
I20260928 02:52:14.474495 30391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:14.476643 30527 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34959
I20260928 02:52:14.476876 30527 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:14.477447 30527 heartbeater.cc:511] Master 127.27.229.254:34959 requested a full tablet report, sending...
I20260928 02:52:14.479564 30320 ts_manager.cc:194] Registered new tserver with Master: 9067a2388744403d99630194b777d289 (127.27.229.193:37367)
I20260928 02:52:14.481115 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:46185
I20260928 02:52:14.483665 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30391
I20260928 02:52:14.483857 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:14.484495 28567 external_mini_cluster.cc:1428] Setting key ae5b8daf31083bb0689f421fd677f857
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:14.527101 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34959
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:14.856698 30531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:14.857031 30531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:14.857215 30531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:14.864717 30531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:14.864879 30531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:14.865042 30531 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:52:14.865134 30531 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:14.865274 30531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:14.875597 30531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:14.876961 30531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:14.878310 30531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:14.892300 30536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:14.892269 30537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:14.892467 30539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:14.893217 30531 server_base.cc:1034] running on GCE node
I20260928 02:52:14.893882 30531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:14.895008 30531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:14.896232 30531 hybrid_clock.cc:648] HybridClock initialized: now 1790563934896167 us; error 62 us; skew 500 ppm
I20260928 02:52:14.896603 30531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:14 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:14.899873 30531 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:14.901429 30531 webserver.cc:460] Webserver started at http://127.27.229.194:39345/ using document root <none> and password file <none>
I20260928 02:52:14.902132 30531 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:14.902262 30531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:14.902606 30531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:14.905797 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "77710a6c95a94d98961a50488ccdc038"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "6ebadd1d69748c0ca9b0cccccb4e9c5f"
server_key_iv: "60ad39c8ecc6bf77ea7431a35904f426"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.906564 30531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "77710a6c95a94d98961a50488ccdc038"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "6ebadd1d69748c0ca9b0cccccb4e9c5f"
server_key_iv: "60ad39c8ecc6bf77ea7431a35904f426"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.911891 30531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260928 02:52:14.915794 30546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:14.917363 30531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:14.917572 30531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "77710a6c95a94d98961a50488ccdc038"
format_stamp: "Formatted at 2026-09-28 02:52:14 on dist-test-slave-s1kg"
server_key: "6ebadd1d69748c0ca9b0cccccb4e9c5f"
server_key_iv: "60ad39c8ecc6bf77ea7431a35904f426"
server_key_version: "encryptionkey@0"
I20260928 02:52:14.917825 30531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:14.947752 30531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:14.952600 30531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:14.952921 30531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:14.955793 30531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:14.955991 30531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:14.956125 30531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:14.956236 30531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:52:14 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563934, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:14.983158 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.956481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:42541 (local address 127.27.229.254:34959)
0928 02:52:14.956691 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:14.956697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:14.958545 (+ 1848us) server_negotiation.cc:407] Connection header received
0928 02:52:14.961250 (+ 2705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:14.961256 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:14.961351 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:14.961481 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:14.964385 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.965203 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.968596 (+ 3393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.969459 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.969587 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.975659 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:14.975666 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:14.975670 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:14.975723 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:14.978460 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.979696 (+ 1236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.979701 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.979704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.979860 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:14.980469 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:14.980474 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:14.980478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:14.980869 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:14.981038 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:14.982242 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0928 02:52:14.982568 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:52:14.983938 30555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.957233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.194:42541)
0928 02:52:14.958321 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:14.958375 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:14.960810 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:14.961819 (+ 1009us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:14.961836 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:14.962929 (+ 1093us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:14.964129 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.964170 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.965397 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.965409 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.968367 (+ 2958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.968388 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.970113 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.970122 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.970489 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.971494 (+ 1005us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:14.971518 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:14.975341 (+ 3823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:14.978624 (+ 3283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.978644 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.978670 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.979531 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.980004 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:14.980009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:14.980013 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:14.980328 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:14.981135 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:14.981145 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:14.981716 (+ 571us) client_negotiation.cc:769] Sending connection context
0928 02:52:14.982088 (+ 372us) client_negotiation.cc:240] Negotiation successful
0928 02:52:14.982571 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":500,"threads_started":1}
I20260928 02:52:15.004667 30612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.989564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.194:46695)
0928 02:52:14.990723 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:14.990748 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:14.990944 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:14.991296 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:14.991301 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:14.991935 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:14.992399 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.992413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.993786 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.993789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.995024 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:14.995037 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.995983 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.995991 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:14.996149 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.996786 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:14.996802 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:14.997490 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:15.001186 (+ 3696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.001194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.001199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.001824 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.002252 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.002258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.002268 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.002466 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.003254 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:15.003259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:15.003506 (+ 247us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.003722 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.004162 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":956,"thread_start_us":779,"threads_started":1}
I20260928 02:52:15.005089 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:14.989593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:46695 (local address 127.27.229.254:34959)
0928 02:52:14.989827 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:14.989836 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:14.990885 (+ 1049us) server_negotiation.cc:407] Connection header received
0928 02:52:14.991081 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:14.991084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:14.991145 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:14.991265 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:14.992626 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.993622 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.995185 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:14.995772 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:14.995908 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:14.997684 (+ 1776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:14.997689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:14.997694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:14.997731 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:15.001044 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.001960 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.001968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.001971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.002112 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.002632 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.002637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.002640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.003140 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:15.003303 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.004289 (+ 986us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.004601 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:15.015470 30531 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:35759
I20260928 02:52:15.015511 30665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:35759 every 8 connection(s)
I20260928 02:52:15.016799 30531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:15.020215 30666 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34959
I20260928 02:52:15.020510 30666 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:15.021294 30666 heartbeater.cc:511] Master 127.27.229.254:34959 requested a full tablet report, sending...
I20260928 02:52:15.023156 30320 ts_manager.cc:194] Registered new tserver with Master: 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759)
I20260928 02:52:15.023998 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:42541
I20260928 02:52:15.027259 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30531
I20260928 02:52:15.027406 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:15.027853 28567 external_mini_cluster.cc:1428] Setting key 4490f737435ea626839ae6e6e164b675
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:15.071065 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34959
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:15.401226 30670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:15.401510 30670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:15.401671 30670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:15.409034 30670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:15.409147 30670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:15.409284 30670 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 02:52:15.409345 30670 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:52:15.409452 30670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:15.419265 30670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:15.420535 30670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:15.421878 30670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:15.435194 30676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:15.435259 30675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:15.435235 30678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:15.435969 30670 server_base.cc:1034] running on GCE node
I20260928 02:52:15.436594 30670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:15.437551 30670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:15.438805 30670 hybrid_clock.cc:648] HybridClock initialized: now 1790563935438748 us; error 49 us; skew 500 ppm
I20260928 02:52:15.439224 30670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:15 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563935, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:15.442461 30670 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:15.444087 30670 webserver.cc:460] Webserver started at http://127.27.229.195:37893/ using document root <none> and password file <none>
I20260928 02:52:15.444801 30670 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:15.444952 30670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:15.445282 30670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:15.448278 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "f586659a71934d38a8a5252f65077be9"
format_stamp: "Formatted at 2026-09-28 02:52:15 on dist-test-slave-s1kg"
server_key: "81b86a17d4a37efa15eba3ed5bd8adc8"
server_key_iv: "20b86cb6a0b7e7bd394ec0418f440b35"
server_key_version: "encryptionkey@0"
I20260928 02:52:15.449179 30670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "f586659a71934d38a8a5252f65077be9"
format_stamp: "Formatted at 2026-09-28 02:52:15 on dist-test-slave-s1kg"
server_key: "81b86a17d4a37efa15eba3ed5bd8adc8"
server_key_iv: "20b86cb6a0b7e7bd394ec0418f440b35"
server_key_version: "encryptionkey@0"
I20260928 02:52:15.454762 30670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:15.458505 30685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:15.460258 30670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:15.460402 30670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "f586659a71934d38a8a5252f65077be9"
format_stamp: "Formatted at 2026-09-28 02:52:15 on dist-test-slave-s1kg"
server_key: "81b86a17d4a37efa15eba3ed5bd8adc8"
server_key_iv: "20b86cb6a0b7e7bd394ec0418f440b35"
server_key_version: "encryptionkey@0"
I20260928 02:52:15.460582 30670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:15.476315 30670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:15.480581 30670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:15.480885 30670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:15.483655 30670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:15.483786 30670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:15.483937 30670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:15.484048 30670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:15.484298 30527 heartbeater.cc:503] Master 127.27.229.254:34959 was elected leader, sending a full tablet report...
Sep 28 02:52:15 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563935, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:15.512018 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.484182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46743 (local address 127.27.229.254:34959)
0928 02:52:15.484418 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.484425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.485947 (+ 1522us) server_negotiation.cc:407] Connection header received
0928 02:52:15.488427 (+ 2480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.488433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.488515 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.488636 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:15.490927 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.491956 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.495529 (+ 3573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.496063 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.496229 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.502599 (+ 6370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:15.502609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:15.502617 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:15.502696 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:15.507109 (+ 4413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.508474 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.508482 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.508488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.508673 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.509373 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.509381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.509385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.509805 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:15.509970 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.511300 (+ 1330us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.511578 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:52:15.513897 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.484838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.195:46743)
0928 02:52:15.485804 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.485848 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.488111 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.488913 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.488929 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.489774 (+ 845us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:15.490710 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.490734 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.492175 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.492183 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.495323 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.495343 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.496363 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.496375 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.496800 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.497977 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:15.498024 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:15.502188 (+ 4164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:15.507323 (+ 5135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.507338 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.507371 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.508285 (+ 914us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.508825 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.508833 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.508840 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.509204 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.509975 (+ 771us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:15.509987 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:15.510652 (+ 665us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.511085 (+ 433us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.511882 (+ 797us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":410,"threads_started":1}
I20260928 02:52:15.528312 30670 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:34129
I20260928 02:52:15.528458 30804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:34129 every 8 connection(s)
I20260928 02:52:15.529999 30670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:15.530805 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30670
I20260928 02:52:15.531018 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:15.531555 28567 external_mini_cluster.cc:1428] Setting key ab92403dfe8954d03fc189c771f287e2
I20260928 02:52:15.532877 30805 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34959
I20260928 02:52:15.533214 30805 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:15.534325 30805 heartbeater.cc:511] Master 127.27.229.254:34959 requested a full tablet report, sending...
I20260928 02:52:15.536060 30320 ts_manager.cc:194] Registered new tserver with Master: f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:15.537060 30320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:46743
I20260928 02:52:15.541834 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.523157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:57709 (local address 127.27.229.254:34959)
0928 02:52:15.523396 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.523406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.527536 (+ 4130us) server_negotiation.cc:407] Connection header received
0928 02:52:15.527674 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.527683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.527789 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.527904 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:15.529199 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.530273 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.532398 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.533060 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.533179 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.535935 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:15.535939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:15.535942 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:15.535972 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:15.539408 (+ 3436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.540207 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.540211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.540213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.540339 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.540700 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.540703 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.540706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.540972 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:15.541087 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.541443 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.541605 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:52:15.541868 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.522735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.27.229.195:57709)
0928 02:52:15.527385 (+ 4650us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.527408 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.527561 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.527967 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.527971 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.528672 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:15.529037 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.529046 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.530443 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.530449 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.532236 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.532251 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.533475 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.533482 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.533694 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.534447 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:15.534473 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:15.535748 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:15.539550 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.539556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.539560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.540053 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.540469 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.540473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.540475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.540597 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.541073 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:15.541076 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:15.541225 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.541330 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.541561 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4477,"thread_start_us":4306,"threads_started":1}
I20260928 02:52:15.547333 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:52:15 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:52:15 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:15.595767 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.581311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34552 (local address 127.27.229.254:34959)
0928 02:52:15.581480 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.581485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.581898 (+ 413us) server_negotiation.cc:407] Connection header received
0928 02:52:15.582048 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.582052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.582115 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.582206 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:15.583759 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.584498 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.585758 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.586218 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.586405 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.590211 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:15.590216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:15.590219 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:15.590253 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:15.592939 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.593729 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.593732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.593735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.593853 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.594214 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.594218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.594220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.594508 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:15.594874 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.595166 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.595390 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:52:15.600071 30320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:15.607025 30320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:15.630877 30819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.619681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.0.0.1:43798)
0928 02:52:15.621596 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.621638 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.621837 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.622863 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.622874 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.622915 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.623426 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.623443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.625391 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.625397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.627628 (+ 2231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.627643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.629870 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.629877 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.630088 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.630145 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.630274 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.630475 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1721,"thread_start_us":2445,"threads_started":1}
I20260928 02:52:15.631817 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.619871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43798 (local address 127.27.229.193:37367)
0928 02:52:15.620427 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.620435 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.621750 (+ 1315us) server_negotiation.cc:407] Connection header received
0928 02:52:15.622041 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.622053 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.622358 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.622568 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.623706 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.625256 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.627823 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.629345 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.629518 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.629719 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.631312 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.631497 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":855,"threads_started":1}
I20260928 02:52:15.632431 30821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.620267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34129 (local address 127.0.0.1:60992)
0928 02:52:15.624155 (+ 3888us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.624172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.624285 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.624990 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.624994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.625025 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.625392 (+ 367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.625404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.627404 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.627409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.629742 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.629758 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.631641 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.631646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.631804 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.631839 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.631915 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.632050 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3785,"mutex_wait_us":1,"thread_start_us":3645,"threads_started":1}
I20260928 02:52:15.632472 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.620289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54598 (local address 127.27.229.194:35759)
0928 02:52:15.623860 (+ 3571us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.623870 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.623901 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:52:15.624003 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.624012 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.624277 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.624450 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.624990 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.626965 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.629857 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.631083 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.631239 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.631387 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.632023 (+ 636us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.632202 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3413,"thread_start_us":3297,"threads_started":1}
I20260928 02:52:15.633054 30818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.619789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35759 (local address 127.0.0.1:54598)
0928 02:52:15.622488 (+ 2699us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.622508 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.622628 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.624435 (+ 1807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.624439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.624462 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.624789 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.624798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.627123 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.627128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.629639 (+ 2511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.629651 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.631495 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.631500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.631722 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.631762 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.631868 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.632720 (+ 852us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"mutex_wait_us":760,"thread_start_us":1484,"threads_started":1}
I20260928 02:52:15.633208 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.620506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60992 (local address 127.27.229.195:34129)
0928 02:52:15.621200 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.621210 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.624280 (+ 3070us) server_negotiation.cc:407] Connection header received
0928 02:52:15.624412 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.624419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.624617 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.624789 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.625684 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.627270 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.629945 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.631443 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.631635 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.631796 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.632709 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.632875 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":393,"threads_started":1}
I20260928 02:52:15.635591 30457 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (DEFAULT_TABLE table=test-table [id=37724d0215f547d0ac4ea0a372627450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:15.636329 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (DEFAULT_TABLE table=test-table [id=37724d0215f547d0ac4ea0a372627450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:15.636853 30457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.637478 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.642536 30596 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (DEFAULT_TABLE table=test-table [id=37724d0215f547d0ac4ea0a372627450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:15.644048 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.649838 30824 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Bootstrap starting.
I20260928 02:52:15.653724 30824 tablet_bootstrap.cc:654] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.654904 30824 log.cc:824] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:15.655627 30825 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Bootstrap starting.
I20260928 02:52:15.659318 30824 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: No bootstrap required, opened a new log
I20260928 02:52:15.659512 30825 tablet_bootstrap.cc:654] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.659783 30824 ts_tablet_manager.cc:1397] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260928 02:52:15.659898 30827 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Bootstrap starting.
I20260928 02:52:15.660885 30825 log.cc:824] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:15.663259 30827 tablet_bootstrap.cc:654] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.664445 30827 log.cc:824] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:15.664772 30825 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: No bootstrap required, opened a new log
I20260928 02:52:15.665058 30825 ts_tablet_manager.cc:1397] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:52:15.669255 30827 tablet_bootstrap.cc:492] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: No bootstrap required, opened a new log
I20260928 02:52:15.669548 30827 ts_tablet_manager.cc:1397] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 02:52:15.670680 30824 raft_consensus.cc:348] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.671180 30824 raft_consensus.cc:374] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.671288 30824 raft_consensus.cc:729] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586659a71934d38a8a5252f65077be9, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.671941 30824 consensus_queue.cc:260] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.675586 30805 heartbeater.cc:503] Master 127.27.229.254:34959 was elected leader, sending a full tablet report...
I20260928 02:52:15.676725 30825 raft_consensus.cc:348] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.677179 30825 raft_consensus.cc:374] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.677277 30825 raft_consensus.cc:729] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067a2388744403d99630194b777d289, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.677855 30824 ts_tablet_manager.cc:1428] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260928 02:52:15.677937 30825 consensus_queue.cc:260] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.678380 30827 raft_consensus.cc:348] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.678854 30827 raft_consensus.cc:374] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.678957 30827 raft_consensus.cc:729] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.679718 30827 consensus_queue.cc:260] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.680029 30825 ts_tablet_manager.cc:1428] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 02:52:15.681171 30666 heartbeater.cc:503] Master 127.27.229.254:34959 was elected leader, sending a full tablet report...
I20260928 02:52:15.681881 30827 ts_tablet_manager.cc:1428] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260928 02:52:15.695916 30832 raft_consensus.cc:482] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:15.696287 30832 raft_consensus.cc:504] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.697968 30832 leader_election.cc:288] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9067a2388744403d99630194b777d289 (127.27.229.193:37367), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:15.705777 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.698466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:34145 (local address 127.27.229.195:34129)
0928 02:52:15.698707 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.698713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.699410 (+ 697us) server_negotiation.cc:407] Connection header received
0928 02:52:15.699611 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.699618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.699693 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.699771 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.700425 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.701211 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.704051 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.704825 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.705025 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.705105 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.705389 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.705514 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:52:15.705942 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.698223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34129 (local address 127.27.229.194:34145)
0928 02:52:15.699291 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.699308 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.699437 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.699817 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.699822 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.699850 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.700275 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.700288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.701454 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.701463 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.703888 (+ 2425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.703901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.705052 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.705059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.705223 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.705248 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.705352 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.705490 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":19,"thread_start_us":739,"threads_started":1}
I20260928 02:52:15.706418 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.698382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:39327 (local address 127.27.229.193:37367)
0928 02:52:15.698638 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.698648 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.699493 (+ 845us) server_negotiation.cc:407] Connection header received
0928 02:52:15.699710 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.699715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.699806 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.699902 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.700676 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.702175 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.704482 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.705350 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.705557 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.705637 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.706009 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.706158 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:52:15.707131 30833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.698252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.27.229.194:39327)
0928 02:52:15.699345 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.699385 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.699541 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.699943 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.699946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.699977 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.700406 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.700417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.702371 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.702378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.704306 (+ 1928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.704319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.705590 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.705597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.705783 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.705807 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.706602 (+ 795us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.706768 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"spinlock_wait_cycles":20352,"thread_start_us":667,"threads_started":1}
I20260928 02:52:15.706974 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7b8f1cd79843cc96f129eb7fa2d1a5" candidate_uuid: "77710a6c95a94d98961a50488ccdc038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9" is_pre_election: true
I20260928 02:52:15.707548 30755 raft_consensus.cc:2454] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77710a6c95a94d98961a50488ccdc038 in term 0.
I20260928 02:52:15.708307 30548 leader_election.cc:302] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:15.708277 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7b8f1cd79843cc96f129eb7fa2d1a5" candidate_uuid: "77710a6c95a94d98961a50488ccdc038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067a2388744403d99630194b777d289" is_pre_election: true
I20260928 02:52:15.708750 30832 raft_consensus.cc:2781] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:15.708786 30482 raft_consensus.cc:2454] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77710a6c95a94d98961a50488ccdc038 in term 0.
I20260928 02:52:15.708957 30832 raft_consensus.cc:482] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:15.709089 30832 raft_consensus.cc:3037] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:15.711608 30832 raft_consensus.cc:504] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.712277 30832 leader_election.cc:288] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [CANDIDATE]: Term 1 election: Requested vote from peers 9067a2388744403d99630194b777d289 (127.27.229.193:37367), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:15.712845 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7b8f1cd79843cc96f129eb7fa2d1a5" candidate_uuid: "77710a6c95a94d98961a50488ccdc038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9"
I20260928 02:52:15.712883 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7b8f1cd79843cc96f129eb7fa2d1a5" candidate_uuid: "77710a6c95a94d98961a50488ccdc038" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067a2388744403d99630194b777d289"
I20260928 02:52:15.713088 30482 raft_consensus.cc:3037] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:15.713100 30755 raft_consensus.cc:3037] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:15.716735 30755 raft_consensus.cc:2454] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77710a6c95a94d98961a50488ccdc038 in term 1.
I20260928 02:52:15.716714 30482 raft_consensus.cc:2454] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77710a6c95a94d98961a50488ccdc038 in term 1.
I20260928 02:52:15.717227 30548 leader_election.cc:302] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:15.717559 30832 raft_consensus.cc:2781] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:15.717993 30832 raft_consensus.cc:686] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 1 LEADER]: Becoming Leader. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Running, Role: LEADER
I20260928 02:52:15.718500 30832 consensus_queue.cc:237] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.723608 30319 catalog_manager.cc:5495] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 reported cstate change: term changed from 0 to 1, leader changed from <none> to 77710a6c95a94d98961a50488ccdc038 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "77710a6c95a94d98961a50488ccdc038" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } health_report { overall_health: UNKNOWN } } }
W20260928 02:52:15.727160 30528 tablet.cc:2369] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:15.769382 30667 tablet.cc:2369] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:52:15 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563933, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:15.783937 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.769451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34959 (local address 127.0.0.1:34560)
0928 02:52:15.770346 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.770368 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.770503 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.770925 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.770929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.771669 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:15.771935 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.771944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.773042 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.773046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.774150 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.774162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.775070 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.775075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.775197 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.775853 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:15.775871 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:15.778102 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:15.781453 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.781464 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.781470 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.781898 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.782234 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:15.782237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:15.782240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:15.782355 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:15.783081 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:15.783088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:15.783236 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.783376 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.783587 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":441,"threads_started":1}
I20260928 02:52:15.783967 30415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.769421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34560 (local address 127.27.229.254:34959)
0928 02:52:15.769673 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.769684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.770459 (+ 775us) server_negotiation.cc:407] Connection header received
0928 02:52:15.770654 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.770658 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.770717 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.770802 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:15.772107 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.772921 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.774311 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.774793 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.774997 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.778414 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:15.778420 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:15.778423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:15.778452 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:15.781305 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.782004 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.782007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.782009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.782128 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:15.782483 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:15.782488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:15.782490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:15.782955 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:15.783090 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.783499 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.783712 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":75}
W20260928 02:52:15.786087 30806 tablet.cc:2369] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:15.789386 30319 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:34560:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:52:15.790522 30319 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:15.797251 30457 tablet_service.cc:1467] Processing CreateTablet for tablet ade4369b6639425fb47cfac2178b4053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a3cd55b14e4b3e91d3b7631034d4ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:15.798082 30457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade4369b6639425fb47cfac2178b4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.797583 30596 tablet_service.cc:1467] Processing CreateTablet for tablet ade4369b6639425fb47cfac2178b4053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a3cd55b14e4b3e91d3b7631034d4ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:15.797585 30735 tablet_service.cc:1467] Processing CreateTablet for tablet ade4369b6639425fb47cfac2178b4053 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=08a3cd55b14e4b3e91d3b7631034d4ff]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:52:15.798425 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade4369b6639425fb47cfac2178b4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.798425 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ade4369b6639425fb47cfac2178b4053. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:15.803499 30825 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: Bootstrap starting.
I20260928 02:52:15.805130 30825 tablet_bootstrap.cc:654] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.805539 30827 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: Bootstrap starting.
I20260928 02:52:15.805539 30824 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: Bootstrap starting.
I20260928 02:52:15.807219 30827 tablet_bootstrap.cc:654] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.807232 30824 tablet_bootstrap.cc:654] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:15.809309 30825 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: No bootstrap required, opened a new log
I20260928 02:52:15.809469 30825 ts_tablet_manager.cc:1397] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 02:52:15.809782 30824 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: No bootstrap required, opened a new log
I20260928 02:52:15.809928 30824 ts_tablet_manager.cc:1397] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:52:15.810191 30827 tablet_bootstrap.cc:492] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: No bootstrap required, opened a new log
I20260928 02:52:15.810344 30827 ts_tablet_manager.cc:1397] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:52:15.810292 30825 raft_consensus.cc:348] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.810554 30825 raft_consensus.cc:374] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.810649 30825 raft_consensus.cc:729] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067a2388744403d99630194b777d289, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.810664 30824 raft_consensus.cc:348] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.810954 30824 raft_consensus.cc:374] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.810905 30825 consensus_queue.cc:260] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.811048 30824 raft_consensus.cc:729] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586659a71934d38a8a5252f65077be9, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.811134 30827 raft_consensus.cc:348] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.811367 30827 raft_consensus.cc:374] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:15.811327 30824 consensus_queue.cc:260] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.811452 30827 raft_consensus.cc:729] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Initialized, Role: FOLLOWER
I20260928 02:52:15.811494 30825 ts_tablet_manager.cc:1428] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:15.811721 30827 consensus_queue.cc:260] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.811803 30830 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:15.811842 30831 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:15.812135 30830 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:15.812201 30831 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:15.812320 30827 ts_tablet_manager.cc:1428] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:15.812856 30832 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:15.813133 30832 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:15.813678 30824 ts_tablet_manager.cc:1428] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:52:15.986923 30831 raft_consensus.cc:482] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:15.987143 30831 raft_consensus.cc:504] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:15.988816 30831 leader_election.cc:288] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:15.997241 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.989331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48169 (local address 127.27.229.194:35759)
0928 02:52:15.989530 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.989536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.990247 (+ 711us) server_negotiation.cc:407] Connection header received
0928 02:52:15.990446 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.990452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.990532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.990643 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.991645 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.992830 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.995155 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.996191 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.996435 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.996524 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.996823 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.996973 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:52:15.997380 30846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.989314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35759 (local address 127.27.229.193:48169)
0928 02:52:15.990146 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.990161 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.990298 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.990671 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.990677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.990715 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.991345 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.991360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.993076 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.993084 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.994928 (+ 1844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.994944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.996436 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.996443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.996618 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.996680 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.996776 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.996935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":170,"spinlock_wait_cycles":21120,"thread_start_us":434,"threads_started":1}
I20260928 02:52:15.997721 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.989416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39061 (local address 127.27.229.195:34129)
0928 02:52:15.989591 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:15.989599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:15.990247 (+ 648us) server_negotiation.cc:407] Connection header received
0928 02:52:15.990447 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:15.990452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:15.990532 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:15.990634 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:15.991645 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.992807 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.995564 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.996578 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.996740 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.996815 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:15.997364 (+ 549us) server_negotiation.cc:299] Negotiation successful
0928 02:52:15.997507 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:52:15.998428 30847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:15.989263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34129 (local address 127.27.229.193:39061)
0928 02:52:15.990070 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:15.990091 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:15.990298 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:15.990671 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:15.990677 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:15.990718 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:15.991353 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.991360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.993179 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.993194 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.995311 (+ 2117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:15.995329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:15.996963 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:15.996969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:15.997173 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:15.997204 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:52:15.997918 (+ 714us) client_negotiation.cc:240] Negotiation successful
0928 02:52:15.998093 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"mutex_wait_us":41,"spinlock_wait_cycles":41472,"thread_start_us":572,"threads_started":1}
I20260928 02:52:15.998454 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade4369b6639425fb47cfac2178b4053" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038" is_pre_election: true
I20260928 02:52:15.998950 30621 raft_consensus.cc:2454] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:15.999054 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade4369b6639425fb47cfac2178b4053" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9" is_pre_election: true
I20260928 02:52:15.999301 30755 raft_consensus.cc:2454] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:15.999658 30410 leader_election.cc:302] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, 9067a2388744403d99630194b777d289; no voters:
I20260928 02:52:16.000025 30831 raft_consensus.cc:2781] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:16.000151 30831 raft_consensus.cc:482] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:16.000245 30831 raft_consensus.cc:3037] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.002365 30831 raft_consensus.cc:504] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.003109 30831 leader_election.cc:288] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Requested vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:16.003647 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade4369b6639425fb47cfac2178b4053" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038"
I20260928 02:52:16.003682 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ade4369b6639425fb47cfac2178b4053" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9"
I20260928 02:52:16.003858 30621 raft_consensus.cc:3037] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.003911 30755 raft_consensus.cc:3037] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.006207 30621 raft_consensus.cc:2454] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:16.006203 30755 raft_consensus.cc:2454] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:16.006685 30408 leader_election.cc:302] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9067a2388744403d99630194b777d289, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:16.007067 30831 raft_consensus.cc:2781] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:16.007524 30831 raft_consensus.cc:686] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [term 1 LEADER]: Becoming Leader. State: Replica: 9067a2388744403d99630194b777d289, State: Running, Role: LEADER
I20260928 02:52:16.008035 30831 consensus_queue.cc:237] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.011399 30848 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: TxnStatusTablet state changed. Reason: New leader 9067a2388744403d99630194b777d289. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.011853 30848 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:16.012686 30850 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:52:16.013042 30850 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:52:16.014137 30318 catalog_manager.cc:5495] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9067a2388744403d99630194b777d289 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:16.059582 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.053200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.27.229.193:37367)
0928 02:52:16.053389 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.053395 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.053420 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:52:16.053510 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.053514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.053588 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.053710 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.054171 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.055157 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.057076 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.057523 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.057624 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.058260 (+ 636us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.058960 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.059076 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.059239 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.059309 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:52:16.059710 30842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.052858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.0.0.1:43804)
0928 02:52:16.053139 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:16.053160 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:16.053311 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:16.053759 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:16.053764 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:16.053778 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:16.054009 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.054022 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.055287 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.055293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.056913 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.056930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.057696 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.057703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.057916 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.057995 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.059073 (+ 1078us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.059106 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:52:16.059207 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:52:16.059382 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:52:16.067615 30755 raft_consensus.cc:1261] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.067607 30621 raft_consensus.cc:1261] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.068562 30848 consensus_queue.cc:1035] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.069367 30831 consensus_queue.cc:1035] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.085994 30832 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: TxnStatusTablet state changed. Reason: New leader 9067a2388744403d99630194b777d289. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.086378 30832 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:16.086776 30830 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: TxnStatusTablet state changed. Reason: New leader 9067a2388744403d99630194b777d289. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.088088 30830 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:16.099478 30830 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.099767 30830 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P f586659a71934d38a8a5252f65077be9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:16.103318 30832 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.103641 30832 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 77710a6c95a94d98961a50488ccdc038: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:52:16.106166 30831 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.106627 30831 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:16.105760 30858 tablet_replica.cc:439] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } } }
I20260928 02:52:16.107000 30858 tablet_replica.cc:442] T ade4369b6639425fb47cfac2178b4053 P 9067a2388744403d99630194b777d289: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:52:16.120878 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.114777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54604 (local address 127.27.229.194:35759)
0928 02:52:16.114999 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.115008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.115417 (+ 409us) server_negotiation.cc:407] Connection header received
0928 02:52:16.115709 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.115713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.115798 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.115952 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.116446 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.117270 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.118684 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.119164 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.119299 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.119714 (+ 415us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.120277 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.120430 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.120492 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.120574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:16.139565 30820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.132683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:40139 (local address 127.27.229.193:37367)
0928 02:52:16.132911 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.132919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.133724 (+ 805us) server_negotiation.cc:407] Connection header received
0928 02:52:16.133923 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.133929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.134032 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.134137 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.134668 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.135884 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.137334 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.138035 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.138177 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.138741 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.138903 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.138995 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.139154 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.139248 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:16.139737 30866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.132723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.27.229.194:40139)
0928 02:52:16.133592 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:16.133616 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:16.133789 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:16.134221 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:16.134225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:16.134236 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:16.134502 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.134510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.136057 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.136062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.137163 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.137177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.138298 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.138306 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.138449 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.138517 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.139011 (+ 494us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.139022 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:52:16.139107 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 02:52:16.139244 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":436,"threads_started":1}
I20260928 02:52:16.149715 30755 raft_consensus.cc:1261] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Refusing update from remote peer 77710a6c95a94d98961a50488ccdc038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.150135 30482 raft_consensus.cc:1261] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Refusing update from remote peer 77710a6c95a94d98961a50488ccdc038: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.150434 30835 consensus_queue.cc:1035] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.150956 30832 consensus_queue.cc:1035] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.194247 30822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.187430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:59463 (local address 127.27.229.194:35759)
0928 02:52:16.187619 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.187624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.187997 (+ 373us) server_negotiation.cc:407] Connection header received
0928 02:52:16.188184 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.188189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.188280 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.188365 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.189119 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.190246 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.192354 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.192927 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.193001 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.193502 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.193637 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.193733 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.193853 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.193953 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:52:16.194480 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.186908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35759 (local address 127.27.229.193:59463)
0928 02:52:16.187888 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:16.187908 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:16.188049 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:16.188531 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:16.188535 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:16.188550 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:16.188968 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.188979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.190378 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.190382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.192193 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.192207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.193152 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.193156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.193267 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.193318 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.193748 (+ 430us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.193759 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:52:16.193831 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 02:52:16.193940 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"spinlock_wait_cycles":3328,"thread_start_us":748,"threads_started":1}
I20260928 02:52:16.232445 30320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:16.232738 30320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:16.236799 30320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Sending DeleteTablet for 3 replicas of tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5
I20260928 02:52:16.238175 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:16 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60992
I20260928 02:52:16.239082 30875 tablet_replica.cc:330] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: stopping tablet replica
I20260928 02:52:16.239095 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:16 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260928 02:52:16.239666 30875 raft_consensus.cc:2229] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:16.240120 30875 raft_consensus.cc:2258] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:16.240891 30320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:16.241578 30320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:16.242435 30875 ts_tablet_manager.cc:1905] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:16.247416 30457 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:16 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:43798
I20260928 02:52:16.249830 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 (DEFAULT_TABLE table=test-table [id=bddf094240574a059ecac1e3ad6c1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:16.249948 30595 tablet_service.cc:1467] Processing CreateTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 (DEFAULT_TABLE table=test-table [id=bddf094240574a059ecac1e3ad6c1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:16.250290 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868d17c5d0db40cf8f8242587007f2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:16.250494 30595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868d17c5d0db40cf8f8242587007f2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:16.254390 30877 tablet_replica.cc:330] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: stopping tablet replica
I20260928 02:52:16.254876 30877 raft_consensus.cc:2229] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:16.255229 30877 raft_consensus.cc:2258] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:16.256908 30457 tablet_service.cc:1467] Processing CreateTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 (DEFAULT_TABLE table=test-table [id=bddf094240574a059ecac1e3ad6c1203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:16.257360 30457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 868d17c5d0db40cf8f8242587007f2f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:16.258386 30827 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Bootstrap starting.
I20260928 02:52:16.260267 30827 tablet_bootstrap.cc:654] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:16.260540 30875 ts_tablet_manager.cc:1918] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:16.260775 30875 log.cc:1196] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/1a7b8f1cd79843cc96f129eb7fa2d1a5
I20260928 02:52:16.261438 30875 ts_tablet_manager.cc:1939] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P f586659a71934d38a8a5252f65077be9: Deleting consensus metadata
I20260928 02:52:16.263052 30877 ts_tablet_manager.cc:1905] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:16.263655 30305 catalog_manager.cc:4841] TS f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129): tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (table test-table [id=37724d0215f547d0ac4ea0a372627450]) successfully deleted
I20260928 02:52:16.271848 30877 ts_tablet_manager.cc:1918] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:16.272174 30877 log.cc:1196] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/1a7b8f1cd79843cc96f129eb7fa2d1a5
I20260928 02:52:16.272886 30877 ts_tablet_manager.cc:1939] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 9067a2388744403d99630194b777d289: Deleting consensus metadata
I20260928 02:52:16.274950 30304 catalog_manager.cc:4841] TS 9067a2388744403d99630194b777d289 (127.27.229.193:37367): tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (table test-table [id=37724d0215f547d0ac4ea0a372627450]) successfully deleted
I20260928 02:52:16.276494 30824 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Bootstrap starting.
I20260928 02:52:16.278445 30824 tablet_bootstrap.cc:654] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:16.281394 30825 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Bootstrap starting.
I20260928 02:52:16.283080 30827 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: No bootstrap required, opened a new log
I20260928 02:52:16.283253 30827 ts_tablet_manager.cc:1397] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.008s
I20260928 02:52:16.283520 30825 tablet_bootstrap.cc:654] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:16.284101 30827 raft_consensus.cc:348] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.284346 30827 raft_consensus.cc:374] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:16.284420 30827 raft_consensus.cc:729] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Initialized, Role: FOLLOWER
I20260928 02:52:16.287228 30827 consensus_queue.cc:260] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.289422 30876 tablet_replica.cc:330] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: stopping tablet replica
I20260928 02:52:16.290470 30876 raft_consensus.cc:2229] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:16.290625 30824 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: No bootstrap required, opened a new log
I20260928 02:52:16.290807 30824 ts_tablet_manager.cc:1397] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.005s
I20260928 02:52:16.291599 30824 raft_consensus.cc:348] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.291852 30824 raft_consensus.cc:374] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:16.291934 30824 raft_consensus.cc:729] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586659a71934d38a8a5252f65077be9, State: Initialized, Role: FOLLOWER
I20260928 02:52:16.292207 30824 consensus_queue.cc:260] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.293104 30876 raft_consensus.cc:2258] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:16.295081 30824 ts_tablet_manager.cc:1428] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260928 02:52:16.295177 30876 ts_tablet_manager.cc:1905] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:16.295571 30827 ts_tablet_manager.cc:1428] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20260928 02:52:16.303293 30876 ts_tablet_manager.cc:1918] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:16.303495 30876 log.cc:1196] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/1a7b8f1cd79843cc96f129eb7fa2d1a5
I20260928 02:52:16.304029 30876 ts_tablet_manager.cc:1939] T 1a7b8f1cd79843cc96f129eb7fa2d1a5 P 77710a6c95a94d98961a50488ccdc038: Deleting consensus metadata
I20260928 02:52:16.304267 30825 tablet_bootstrap.cc:492] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: No bootstrap required, opened a new log
I20260928 02:52:16.304445 30825 ts_tablet_manager.cc:1397] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20260928 02:52:16.305253 30825 raft_consensus.cc:348] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.305519 30825 raft_consensus.cc:374] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:16.305622 30825 raft_consensus.cc:729] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067a2388744403d99630194b777d289, State: Initialized, Role: FOLLOWER
I20260928 02:52:16.305855 30307 catalog_manager.cc:4841] TS 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759): tablet 1a7b8f1cd79843cc96f129eb7fa2d1a5 (table test-table [id=37724d0215f547d0ac4ea0a372627450]) successfully deleted
I20260928 02:52:16.305900 30825 consensus_queue.cc:260] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.306505 30825 ts_tablet_manager.cc:1428] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260928 02:52:16.585873 30858 raft_consensus.cc:482] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:16.586107 30858 raft_consensus.cc:504] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.586849 30858 leader_election.cc:288] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:16.587476 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868d17c5d0db40cf8f8242587007f2f8" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038" is_pre_election: true
I20260928 02:52:16.587509 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868d17c5d0db40cf8f8242587007f2f8" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9" is_pre_election: true
I20260928 02:52:16.587738 30621 raft_consensus.cc:2454] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:16.587795 30755 raft_consensus.cc:2454] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:16.588253 30410 leader_election.cc:302] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, 9067a2388744403d99630194b777d289; no voters:
I20260928 02:52:16.588533 30858 raft_consensus.cc:2781] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:16.588686 30858 raft_consensus.cc:482] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:16.588810 30858 raft_consensus.cc:3037] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.590790 30858 raft_consensus.cc:504] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.591481 30858 leader_election.cc:288] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Requested vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:16.592013 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868d17c5d0db40cf8f8242587007f2f8" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038"
I20260928 02:52:16.592037 30755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "868d17c5d0db40cf8f8242587007f2f8" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9"
I20260928 02:52:16.592229 30621 raft_consensus.cc:3037] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.592253 30755 raft_consensus.cc:3037] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:16.594770 30621 raft_consensus.cc:2454] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:16.595301 30410 leader_election.cc:302] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, 9067a2388744403d99630194b777d289; no voters:
I20260928 02:52:16.595409 30755 raft_consensus.cc:2454] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:16.595630 30858 raft_consensus.cc:2781] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:16.595861 30858 raft_consensus.cc:686] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 1 LEADER]: Becoming Leader. State: Replica: 9067a2388744403d99630194b777d289, State: Running, Role: LEADER
I20260928 02:52:16.596172 30858 consensus_queue.cc:237] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:16.599030 30320 catalog_manager.cc:5495] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9067a2388744403d99630194b777d289 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:16.664325 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.650813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.27.229.193:37367)
0928 02:52:16.651302 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.651306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.659550 (+ 8244us) server_negotiation.cc:407] Connection header received
0928 02:52:16.659806 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.659811 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.659882 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.659991 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.660420 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.661164 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.662527 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.663025 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.663156 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.663563 (+ 407us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.663670 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.663767 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.663909 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.664010 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":237,"threads_started":1}
I20260928 02:52:16.678292 30883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.673013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:60745 (local address 127.27.229.193:37367)
0928 02:52:16.673196 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:16.673202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:16.673297 (+ 95us) server_negotiation.cc:407] Connection header received
0928 02:52:16.673495 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:16.673504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:16.673585 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:16.673682 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:16.674075 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.674802 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.676629 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.677015 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.677131 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.677600 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.677717 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.677789 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:16.677932 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:52:16.677992 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":74}
I20260928 02:52:16.678404 30872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:16.673000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.27.229.193:60745)
0928 02:52:16.673198 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:16.673219 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:16.673368 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:16.673704 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:16.673708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:16.673719 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:16.673954 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.673963 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.674992 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.674998 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.676468 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:16.676483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:16.677177 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:16.677183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:16.677355 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:16.677403 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:16.677818 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:16.677828 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:52:16.677908 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:52:16.678049 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":35}
I20260928 02:52:16.687372 30755 raft_consensus.cc:1261] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.687439 30621 raft_consensus.cc:1261] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:16.687865 30858 consensus_queue.cc:1035] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.688155 30848 consensus_queue.cc:1035] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:16.693971 30870 mvcc.cc:205] Tried to move back new op lower bound from 7334149884666441728 to 7334149884300468224. Current Snapshot: MvccSnapshot[applied={T|T < 7334149884666441728}]
W20260928 02:52:16.707985 30870 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:52:18.299466 30318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:18.299762 30318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:18.304144 30318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Sending DeleteTablet for 3 replicas of tablet 868d17c5d0db40cf8f8242587007f2f8
I20260928 02:52:18.305138 30457 tablet_service.cc:1514] Processing DeleteTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:43798
I20260928 02:52:18.305137 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260928 02:52:18.305197 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 868d17c5d0db40cf8f8242587007f2f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60992
I20260928 02:52:18.312788 30910 tablet_replica.cc:330] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: stopping tablet replica
I20260928 02:52:18.313052 30910 raft_consensus.cc:2229] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:18.313347 30910 raft_consensus.cc:2258] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.313973 30908 tablet_replica.cc:330] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: stopping tablet replica
I20260928 02:52:18.314213 30908 raft_consensus.cc:2229] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:18.314253 30909 tablet_replica.cc:330] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: stopping tablet replica
I20260928 02:52:18.314482 30909 raft_consensus.cc:2229] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:18.314541 30910 ts_tablet_manager.cc:1905] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.314731 30909 raft_consensus.cc:2258] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.314805 30908 raft_consensus.cc:2258] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.315827 30908 ts_tablet_manager.cc:1905] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.315884 30909 ts_tablet_manager.cc:1905] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.321552 30910 ts_tablet_manager.cc:1918] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:18.321789 30910 log.cc:1196] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/868d17c5d0db40cf8f8242587007f2f8
I20260928 02:52:18.322366 30910 ts_tablet_manager.cc:1939] T 868d17c5d0db40cf8f8242587007f2f8 P f586659a71934d38a8a5252f65077be9: Deleting consensus metadata
I20260928 02:52:18.322531 30908 ts_tablet_manager.cc:1918] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:18.322767 30908 log.cc:1196] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/868d17c5d0db40cf8f8242587007f2f8
I20260928 02:52:18.323045 30909 ts_tablet_manager.cc:1918] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:18.323239 30909 log.cc:1196] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/868d17c5d0db40cf8f8242587007f2f8
I20260928 02:52:18.323282 30908 ts_tablet_manager.cc:1939] T 868d17c5d0db40cf8f8242587007f2f8 P 9067a2388744403d99630194b777d289: Deleting consensus metadata
I20260928 02:52:18.323731 30909 ts_tablet_manager.cc:1939] T 868d17c5d0db40cf8f8242587007f2f8 P 77710a6c95a94d98961a50488ccdc038: Deleting consensus metadata
I20260928 02:52:18.323968 30305 catalog_manager.cc:4841] TS f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129): tablet 868d17c5d0db40cf8f8242587007f2f8 (table test-table [id=bddf094240574a059ecac1e3ad6c1203]) successfully deleted
I20260928 02:52:18.324381 30304 catalog_manager.cc:4841] TS 9067a2388744403d99630194b777d289 (127.27.229.193:37367): tablet 868d17c5d0db40cf8f8242587007f2f8 (table test-table [id=bddf094240574a059ecac1e3ad6c1203]) successfully deleted
I20260928 02:52:18.324873 30307 catalog_manager.cc:4841] TS 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759): tablet 868d17c5d0db40cf8f8242587007f2f8 (table test-table [id=bddf094240574a059ecac1e3ad6c1203]) successfully deleted
Sep 28 02:52:18 dist-test-slave-s1kg krb5kdc[30275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:52:18 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:18.362495 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.346672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36950 (local address 127.27.229.254:34959)
0928 02:52:18.347286 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.347290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.347313 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:18.347369 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.347374 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.347431 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.347533 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:18.348990 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.349953 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.351442 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.351862 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.351971 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.356227 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:18.356233 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:18.356236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:18.356275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:18.359818 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:18.360548 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:18.360552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:18.360554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:18.360694 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:18.361115 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:18.361119 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:18.361121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:18.361440 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:18.361634 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.362004 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.362203 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":387,"threads_started":1}
I20260928 02:52:18.366407 30318 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:18.367151 30318 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:18.374365 30457 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3ad2109493414593d5712bfbd906d4 (DEFAULT_TABLE table=test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.374598 30596 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3ad2109493414593d5712bfbd906d4 (DEFAULT_TABLE table=test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.374675 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 6e3ad2109493414593d5712bfbd906d4 (DEFAULT_TABLE table=test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.374869 30457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3ad2109493414593d5712bfbd906d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.375056 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3ad2109493414593d5712bfbd906d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.375128 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e3ad2109493414593d5712bfbd906d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.380501 30919 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Bootstrap starting.
I20260928 02:52:18.381829 30920 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Bootstrap starting.
I20260928 02:52:18.382377 30921 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Bootstrap starting.
I20260928 02:52:18.383375 30919 tablet_bootstrap.cc:654] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.384119 30921 tablet_bootstrap.cc:654] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.385612 30919 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: No bootstrap required, opened a new log
I20260928 02:52:18.385839 30919 ts_tablet_manager.cc:1397] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 02:52:18.386559 30920 tablet_bootstrap.cc:654] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.386660 30919 raft_consensus.cc:348] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.386960 30919 raft_consensus.cc:374] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.387115 30919 raft_consensus.cc:729] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586659a71934d38a8a5252f65077be9, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.387398 30919 consensus_queue.cc:260] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.388266 30919 ts_tablet_manager.cc:1428] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:18.388650 30921 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: No bootstrap required, opened a new log
I20260928 02:52:18.388875 30921 ts_tablet_manager.cc:1397] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260928 02:52:18.390383 30921 raft_consensus.cc:348] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.390699 30921 raft_consensus.cc:374] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.390811 30921 raft_consensus.cc:729] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067a2388744403d99630194b777d289, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.391219 30921 consensus_queue.cc:260] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.392755 30921 ts_tablet_manager.cc:1428] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260928 02:52:18.394284 30920 tablet_bootstrap.cc:492] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: No bootstrap required, opened a new log
I20260928 02:52:18.394642 30920 ts_tablet_manager.cc:1397] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260928 02:52:18.395365 30920 raft_consensus.cc:348] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.395577 30920 raft_consensus.cc:374] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.395690 30920 raft_consensus.cc:729] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.395922 30920 consensus_queue.cc:260] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.396647 30920 ts_tablet_manager.cc:1428] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:18.407863 30925 raft_consensus.cc:482] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:18.408145 30925 raft_consensus.cc:504] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.409720 30925 leader_election.cc:288] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9067a2388744403d99630194b777d289 (127.27.229.193:37367), 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759)
I20260928 02:52:18.433408 30930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.410256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:57225 (local address 127.27.229.194:35759)
0928 02:52:18.426158 (+ 15902us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.426164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.426187 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:18.426268 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.426272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.426347 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.426469 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:18.427172 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.428301 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.430896 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.432214 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.432394 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.432469 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.432893 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.433060 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15752,"thread_start_us":15575,"threads_started":1}
I20260928 02:52:18.433425 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.410058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35759 (local address 127.27.229.195:57225)
0928 02:52:18.425161 (+ 15103us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:18.425193 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:18.425379 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:18.426493 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:18.426502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:18.426529 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:18.426973 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.426983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.428434 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.428445 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.430680 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.430688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.432463 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.432470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.432708 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.432742 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:52:18.432871 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:52:18.433008 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14870,"mutex_wait_us":57,"thread_start_us":15385,"threads_started":1}
I20260928 02:52:18.433570 30928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.410058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.27.229.195:47857)
0928 02:52:18.425662 (+ 15604us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:18.425689 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:18.425861 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:18.426225 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:18.426229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:18.426252 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:18.426532 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.426540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.428266 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.428270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.430671 (+ 2401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.430688 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.432655 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.432661 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.432866 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.432898 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:52:18.432984 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 02:52:18.433115 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15380,"mutex_wait_us":149,"thread_start_us":15500,"threads_started":1}
I20260928 02:52:18.434235 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.410237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:47857 (local address 127.27.229.193:37367)
0928 02:52:18.425599 (+ 15362us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.425605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.425885 (+ 280us) server_negotiation.cc:407] Connection header received
0928 02:52:18.425969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.425974 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.426053 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.426158 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:18.426786 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.428072 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.430896 (+ 2824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.432218 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.432397 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.432469 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.433783 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.433923 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15204,"thread_start_us":15364,"threads_started":1}
I20260928 02:52:18.434505 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3ad2109493414593d5712bfbd906d4" candidate_uuid: "f586659a71934d38a8a5252f65077be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038" is_pre_election: true
I20260928 02:52:18.434863 30620 raft_consensus.cc:2454] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f586659a71934d38a8a5252f65077be9 in term 0.
I20260928 02:52:18.434876 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3ad2109493414593d5712bfbd906d4" candidate_uuid: "f586659a71934d38a8a5252f65077be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067a2388744403d99630194b777d289" is_pre_election: true
I20260928 02:52:18.435134 30482 raft_consensus.cc:2454] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f586659a71934d38a8a5252f65077be9 in term 0.
I20260928 02:52:18.435473 30689 leader_election.cc:302] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:18.435936 30925 raft_consensus.cc:2781] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:18.436070 30925 raft_consensus.cc:482] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:18.436174 30925 raft_consensus.cc:3037] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.438220 30925 raft_consensus.cc:504] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.438799 30925 leader_election.cc:288] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [CANDIDATE]: Term 1 election: Requested vote from peers 9067a2388744403d99630194b777d289 (127.27.229.193:37367), 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759)
I20260928 02:52:18.439373 30620 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3ad2109493414593d5712bfbd906d4" candidate_uuid: "f586659a71934d38a8a5252f65077be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038"
I20260928 02:52:18.439376 30482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e3ad2109493414593d5712bfbd906d4" candidate_uuid: "f586659a71934d38a8a5252f65077be9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9067a2388744403d99630194b777d289"
I20260928 02:52:18.439585 30620 raft_consensus.cc:3037] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.439585 30482 raft_consensus.cc:3037] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.442523 30620 raft_consensus.cc:2454] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f586659a71934d38a8a5252f65077be9 in term 1.
I20260928 02:52:18.442523 30482 raft_consensus.cc:2454] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f586659a71934d38a8a5252f65077be9 in term 1.
I20260928 02:52:18.443087 30686 leader_election.cc:302] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, 9067a2388744403d99630194b777d289, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:18.443498 30925 raft_consensus.cc:2781] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:18.444018 30925 raft_consensus.cc:686] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 1 LEADER]: Becoming Leader. State: Replica: f586659a71934d38a8a5252f65077be9, State: Running, Role: LEADER
I20260928 02:52:18.444401 30925 consensus_queue.cc:237] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.447844 30320 catalog_manager.cc:5495] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f586659a71934d38a8a5252f65077be9 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "f586659a71934d38a8a5252f65077be9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } health_report { overall_health: HEALTHY } } }
I20260928 02:52:18.482877 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.476744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.27.229.195:34129)
0928 02:52:18.477299 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.477303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.477322 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:52:18.477402 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.477406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.477477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.477662 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:18.478099 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.479054 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.480482 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.480920 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.481061 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.481546 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.482167 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.482326 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.482405 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.482485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":288,"threads_started":1}
I20260928 02:52:18.499909 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.493899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:59297 (local address 127.27.229.193:37367)
0928 02:52:18.494159 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.494169 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.495066 (+ 897us) server_negotiation.cc:407] Connection header received
0928 02:52:18.495304 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.495309 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.495380 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.495475 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:18.495993 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.496740 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.498122 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.498535 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.498679 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.499192 (+ 513us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.499295 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.499393 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.499529 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.499607 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:52:18.500021 30937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.493829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37367 (local address 127.27.229.195:59297)
0928 02:52:18.494804 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:18.494820 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:18.495158 (+ 338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:18.495502 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:18.495507 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:18.495518 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:18.495820 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.495828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.496888 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.496893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.497970 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.497983 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.498724 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.498733 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.498945 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.499008 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.499392 (+ 384us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.499405 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:52:18.499509 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:52:18.499649 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"thread_start_us":562,"threads_started":1}
I20260928 02:52:18.508409 30620 raft_consensus.cc:1261] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Refusing update from remote peer f586659a71934d38a8a5252f65077be9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:18.508455 30482 raft_consensus.cc:1261] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Refusing update from remote peer f586659a71934d38a8a5252f65077be9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:18.509199 30931 consensus_queue.cc:1035] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:18.509625 30925 consensus_queue.cc:1035] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:18.514262 30892 mvcc.cc:205] Tried to move back new op lower bound from 7334149892124405760 to 7334149891871911936. Current Snapshot: MvccSnapshot[applied={T|T < 7334149892124405760}]
I20260928 02:52:18.550109 30934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.537821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:60793 (local address 127.27.229.195:34129)
0928 02:52:18.537977 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.537983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.543433 (+ 5450us) server_negotiation.cc:407] Connection header received
0928 02:52:18.543651 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.543657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.543744 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.543866 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:18.544479 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.545607 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.547885 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.548609 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.548757 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.549314 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.549417 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.549523 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.549687 (+ 164us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.549781 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 02:52:18.550794 30943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.537380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34129 (local address 127.27.229.193:60793)
0928 02:52:18.543306 (+ 5926us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:18.543327 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:18.543470 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:18.543883 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:18.543888 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:18.543899 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:52:18.544307 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.544321 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.545780 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.545786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.547422 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:18.547446 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.548820 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.548828 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:18.549087 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.549138 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.549540 (+ 402us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.549553 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:52:18.550389 (+ 836us) client_negotiation.cc:240] Negotiation successful
0928 02:52:18.550528 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5763,"thread_start_us":5625,"threads_started":1}
I20260928 02:52:18.585210 30320 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:18.585464 30320 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:18.588508 30320 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Sending DeleteTablet for 3 replicas of tablet 6e3ad2109493414593d5712bfbd906d4
I20260928 02:52:18.589430 30457 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3ad2109493414593d5712bfbd906d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:43798
I20260928 02:52:18.589424 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3ad2109493414593d5712bfbd906d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260928 02:52:18.589715 30909 tablet_replica.cc:330] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: stopping tablet replica
I20260928 02:52:18.589720 30908 tablet_replica.cc:330] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: stopping tablet replica
I20260928 02:52:18.589751 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e3ad2109493414593d5712bfbd906d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:18 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60992
I20260928 02:52:18.589905 30909 raft_consensus.cc:2229] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:18.589947 30908 raft_consensus.cc:2229] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:18.590009 30910 tablet_replica.cc:330] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: stopping tablet replica
I20260928 02:52:18.590113 30909 raft_consensus.cc:2258] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.590140 30908 raft_consensus.cc:2258] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.590179 30910 raft_consensus.cc:2229] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:18.590682 30910 raft_consensus.cc:2258] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:18.591301 30909 ts_tablet_manager.cc:1905] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.591301 30908 ts_tablet_manager.cc:1905] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.591850 30910 ts_tablet_manager.cc:1905] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:18.598048 30909 ts_tablet_manager.cc:1918] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:18.598048 30908 ts_tablet_manager.cc:1918] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:18.598218 30909 log.cc:1196] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/6e3ad2109493414593d5712bfbd906d4
I20260928 02:52:18.598652 30909 ts_tablet_manager.cc:1939] T 6e3ad2109493414593d5712bfbd906d4 P 77710a6c95a94d98961a50488ccdc038: Deleting consensus metadata
I20260928 02:52:18.599264 30910 ts_tablet_manager.cc:1918] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:52:18.599426 30910 log.cc:1196] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/6e3ad2109493414593d5712bfbd906d4
I20260928 02:52:18.599785 30307 catalog_manager.cc:4841] TS 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759): tablet 6e3ad2109493414593d5712bfbd906d4 (table test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]) successfully deleted
I20260928 02:52:18.599851 30910 ts_tablet_manager.cc:1939] T 6e3ad2109493414593d5712bfbd906d4 P f586659a71934d38a8a5252f65077be9: Deleting consensus metadata
I20260928 02:52:18.599997 30320 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 02:52:18.600363 30908 log.cc:1196] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/6e3ad2109493414593d5712bfbd906d4
I20260928 02:52:18.600811 30908 ts_tablet_manager.cc:1939] T 6e3ad2109493414593d5712bfbd906d4 P 9067a2388744403d99630194b777d289: Deleting consensus metadata
I20260928 02:52:18.600980 30305 catalog_manager.cc:4841] TS f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129): tablet 6e3ad2109493414593d5712bfbd906d4 (table test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]) successfully deleted
W20260928 02:52:18.601070 30320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:18.601979 30304 catalog_manager.cc:4841] TS 9067a2388744403d99630194b777d289 (127.27.229.193:37367): tablet 6e3ad2109493414593d5712bfbd906d4 (table test-table [id=e339a007c7c84f9c9378b6a0c5c68c41]) successfully deleted
I20260928 02:52:18.608613 30457 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (DEFAULT_TABLE table=test-table [id=38e8296a55504c1083300dd067124588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.609102 30457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5d8fe24e84fecaa93c41f6de4cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.609077 30735 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (DEFAULT_TABLE table=test-table [id=38e8296a55504c1083300dd067124588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.609524 30735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5d8fe24e84fecaa93c41f6de4cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.613588 30921 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Bootstrap starting.
I20260928 02:52:18.615253 30921 tablet_bootstrap.cc:654] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.615290 30596 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (DEFAULT_TABLE table=test-table [id=38e8296a55504c1083300dd067124588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:18.615744 30596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5d8fe24e84fecaa93c41f6de4cd35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:18.617390 30921 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: No bootstrap required, opened a new log
I20260928 02:52:18.617623 30921 ts_tablet_manager.cc:1397] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260928 02:52:18.618454 30921 raft_consensus.cc:348] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.618705 30921 raft_consensus.cc:374] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.618790 30921 raft_consensus.cc:729] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9067a2388744403d99630194b777d289, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.619081 30921 consensus_queue.cc:260] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.619647 30921 ts_tablet_manager.cc:1428] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:18.621738 30919 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Bootstrap starting.
I20260928 02:52:18.623445 30920 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Bootstrap starting.
I20260928 02:52:18.624267 30919 tablet_bootstrap.cc:654] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.625092 30920 tablet_bootstrap.cc:654] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:18.627193 30919 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: No bootstrap required, opened a new log
I20260928 02:52:18.627441 30919 ts_tablet_manager.cc:1397] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 02:52:18.627525 30920 tablet_bootstrap.cc:492] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: No bootstrap required, opened a new log
I20260928 02:52:18.627656 30920 ts_tablet_manager.cc:1397] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260928 02:52:18.628190 30920 raft_consensus.cc:348] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.628383 30920 raft_consensus.cc:374] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.628453 30920 raft_consensus.cc:729] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77710a6c95a94d98961a50488ccdc038, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.628758 30920 consensus_queue.cc:260] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.628619 30919 raft_consensus.cc:348] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.629204 30920 ts_tablet_manager.cc:1428] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 02:52:18.629256 30919 raft_consensus.cc:374] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:18.629429 30919 raft_consensus.cc:729] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f586659a71934d38a8a5252f65077be9, State: Initialized, Role: FOLLOWER
I20260928 02:52:18.629719 30919 consensus_queue.cc:260] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.630283 30919 ts_tablet_manager.cc:1428] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 02:52:18.827720 30900 raft_consensus.cc:482] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:18.827983 30900 raft_consensus.cc:504] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.828733 30900 leader_election.cc:288] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:18.829389 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5d8fe24e84fecaa93c41f6de4cd35" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038" is_pre_election: true
I20260928 02:52:18.829445 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5d8fe24e84fecaa93c41f6de4cd35" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9" is_pre_election: true
I20260928 02:52:18.829669 30621 raft_consensus.cc:2454] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:18.829736 30754 raft_consensus.cc:2454] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 0.
I20260928 02:52:18.830224 30408 leader_election.cc:302] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9067a2388744403d99630194b777d289, f586659a71934d38a8a5252f65077be9; no voters:
I20260928 02:52:18.830513 30900 raft_consensus.cc:2781] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:18.830772 30900 raft_consensus.cc:482] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:18.831018 30900 raft_consensus.cc:3037] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.833283 30900 raft_consensus.cc:504] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.834120 30900 leader_election.cc:288] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Requested vote from peers 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759), f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129)
I20260928 02:52:18.834676 30754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5d8fe24e84fecaa93c41f6de4cd35" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f586659a71934d38a8a5252f65077be9"
I20260928 02:52:18.834800 30621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5d8fe24e84fecaa93c41f6de4cd35" candidate_uuid: "9067a2388744403d99630194b777d289" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77710a6c95a94d98961a50488ccdc038"
I20260928 02:52:18.834982 30754 raft_consensus.cc:3037] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.835043 30621 raft_consensus.cc:3037] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:18.838371 30621 raft_consensus.cc:2454] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:18.839059 30754 raft_consensus.cc:2454] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9067a2388744403d99630194b777d289 in term 1.
I20260928 02:52:18.839126 30410 leader_election.cc:302] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77710a6c95a94d98961a50488ccdc038, 9067a2388744403d99630194b777d289; no voters:
I20260928 02:52:18.839555 30900 raft_consensus.cc:2781] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:18.840095 30900 raft_consensus.cc:686] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 1 LEADER]: Becoming Leader. State: Replica: 9067a2388744403d99630194b777d289, State: Running, Role: LEADER
I20260928 02:52:18.840415 30900 consensus_queue.cc:237] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } }
I20260928 02:52:18.843641 30318 catalog_manager.cc:5495] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9067a2388744403d99630194b777d289 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "9067a2388744403d99630194b777d289" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9067a2388744403d99630194b777d289" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:18.879704 30929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:18.874008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.27.229.193:37367)
0928 02:52:18.874218 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:18.874227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:18.874391 (+ 164us) server_negotiation.cc:407] Connection header received
0928 02:52:18.874516 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:18.874521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:18.874626 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:18.874733 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:18.875174 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.876338 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.877598 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:18.878261 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:18.878408 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:18.878896 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:18.879033 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:18.879144 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:18.879290 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:52:18.879385 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:52:18.890913 30620 raft_consensus.cc:1261] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:18.891219 30754 raft_consensus.cc:1261] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Refusing update from remote peer 9067a2388744403d99630194b777d289: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:18.891484 30900 consensus_queue.cc:1035] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77710a6c95a94d98961a50488ccdc038" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:18.891819 30891 consensus_queue.cc:1035] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f586659a71934d38a8a5252f65077be9" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:18.901087 30893 mvcc.cc:205] Tried to move back new op lower bound from 7334149893690740736 to 7334149893493121024. Current Snapshot: MvccSnapshot[applied={T|T < 7334149893690740736}]
W20260928 02:52:18.918936 30889 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 02:52:20.514714 30318 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:20.515065 30318 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:20.519505 30318 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d8739cf2ff040a5afe2deaabe272395: Sending DeleteTablet for 3 replicas of tablet 9df5d8fe24e84fecaa93c41f6de4cd35
I20260928 02:52:20.520339 30456 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:43798
I20260928 02:52:20.520439 30596 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:54598
I20260928 02:52:20.520470 30735 tablet_service.cc:1514] Processing DeleteTablet for tablet 9df5d8fe24e84fecaa93c41f6de4cd35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60992
I20260928 02:52:20.522439 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30391
I20260928 02:52:20.548638 30972 tablet_replica.cc:330] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: stopping tablet replica
I20260928 02:52:20.548795 30974 tablet_replica.cc:330] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: stopping tablet replica
I20260928 02:52:20.548974 30972 raft_consensus.cc:2229] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:20.549510 30972 raft_consensus.cc:2258] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:20.549669 30974 raft_consensus.cc:2229] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:20.550031 30974 raft_consensus.cc:2258] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:20.550892 30972 ts_tablet_manager.cc:1905] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:20.553531 30973 tablet_replica.cc:330] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: stopping tablet replica
I20260928 02:52:20.553761 30973 raft_consensus.cc:2229] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:20.554047 30973 raft_consensus.cc:2258] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:20.554287 30974 ts_tablet_manager.cc:1905] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:20.555059 30973 ts_tablet_manager.cc:1905] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:20.558182 30972 ts_tablet_manager.cc:1918] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:20.558359 30972 log.cc:1196] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/9df5d8fe24e84fecaa93c41f6de4cd35
I20260928 02:52:20.559051 30972 ts_tablet_manager.cc:1939] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 9067a2388744403d99630194b777d289: Deleting consensus metadata
I20260928 02:52:20.560236 30304 catalog_manager.cc:4841] TS 9067a2388744403d99630194b777d289 (127.27.229.193:37367): tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (table test-table [id=38e8296a55504c1083300dd067124588]) successfully deleted
I20260928 02:52:20.561388 30974 ts_tablet_manager.cc:1918] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:20.561614 30974 log.cc:1196] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/9df5d8fe24e84fecaa93c41f6de4cd35
I20260928 02:52:20.562106 30973 ts_tablet_manager.cc:1918] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 02:52:20.562172 30974 ts_tablet_manager.cc:1939] T 9df5d8fe24e84fecaa93c41f6de4cd35 P f586659a71934d38a8a5252f65077be9: Deleting consensus metadata
I20260928 02:52:20.562326 30973 log.cc:1196] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/9df5d8fe24e84fecaa93c41f6de4cd35
I20260928 02:52:20.562809 30973 ts_tablet_manager.cc:1939] T 9df5d8fe24e84fecaa93c41f6de4cd35 P 77710a6c95a94d98961a50488ccdc038: Deleting consensus metadata
I20260928 02:52:20.563331 30305 catalog_manager.cc:4841] TS f586659a71934d38a8a5252f65077be9 (127.27.229.195:34129): tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (table test-table [id=38e8296a55504c1083300dd067124588]) successfully deleted
I20260928 02:52:20.564219 30307 catalog_manager.cc:4841] TS 77710a6c95a94d98961a50488ccdc038 (127.27.229.194:35759): tablet 9df5d8fe24e84fecaa93c41f6de4cd35 (table test-table [id=38e8296a55504c1083300dd067124588]) successfully deleted
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:20.578505 30975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:20.523596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49906 (local address 127.27.229.193:37367)
0928 02:52:20.548085 (+ 24489us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:20.548095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:20.549673 (+ 1578us) server_negotiation.cc:407] Connection header received
0928 02:52:20.549752 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:20.549757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:20.549845 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:20.549948 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:20.553039 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.560743 (+ 7704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.562902 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.563655 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.563784 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:20.572542 (+ 8758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:20.572559 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:20.572577 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:20.572617 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:20.575839 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.576650 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.576658 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.576660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.576838 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.577178 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.577181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.577183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.577532 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:20.577737 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:20.578070 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:52:20.578237 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24266,"thread_start_us":24533,"threads_started":1}
W20260928 02:52:20.579254 30522 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49906
W20260928 02:52:20.579958 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:20.580155 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30391
I20260928 02:52:20.606208 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30531
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:20.622054 30977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:20.607051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51954 (local address 127.27.229.194:35759)
0928 02:52:20.607566 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:20.607571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:20.607587 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:52:20.607639 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:20.607643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:20.607702 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:20.607796 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:20.609067 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.609795 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.611181 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.611595 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.611743 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:20.616105 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:20.616121 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:20.616137 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:20.616181 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:20.619326 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.620052 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.620060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.620062 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.620202 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.620542 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.620545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.620547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.620933 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:20.621172 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:20.621558 (+ 386us) server_negotiation.cc:299] Negotiation successful
0928 02:52:20.621774 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":284,"threads_started":1}
W20260928 02:52:20.622948 30661 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51954
W20260928 02:52:20.623632 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30531: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:20.623811 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30531
I20260928 02:52:20.645645 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30670
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563938, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:20.661158 30978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:20.646433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48816 (local address 127.27.229.195:34129)
0928 02:52:20.646993 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:20.646998 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:20.647014 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:52:20.647067 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:20.647097 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:20.647155 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:20.647253 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:20.648516 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.649257 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.650442 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:20.650918 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:20.651042 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:20.655357 (+ 4315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:20.655373 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:20.655390 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:20.655428 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:20.658283 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.659065 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.659072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.659075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.659203 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:20.659591 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:20.659595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:20.659597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:20.659983 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:20.660252 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:20.660568 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 02:52:20.660750 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":258,"threads_started":1}
W20260928 02:52:20.661907 30799 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48816
W20260928 02:52:20.662518 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:20.662730 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30670
I20260928 02:52:20.685696 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30288
I20260928 02:52:20.687050 30360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:20.802248 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30288
2026-09-28T02:52:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9699 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30982](info): set up 2 sockets
Sep 28 02:52:20 dist-test-slave-s1kg krb5kdc[30982](info): commencing operation
krb5kdc: starting...
W20260928 02:52:21.915160 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:22 dist-test-slave-s1kg krb5kdc[30982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:22.115216 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34803
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:22.443676 30995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:22.443994 30995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:22.444154 30995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:22.451754 30995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:22.451880 30995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:22.451956 30995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:22.452049 30995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:22.452135 30995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:22.462167 30995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:22.463593 30995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:22.464905 30995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:22.477263 31003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:22.477375 31000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:22.477375 31001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:22.477682 30995 server_base.cc:1034] running on GCE node
I20260928 02:52:22.478698 30995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:22.480142 30995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:22.481362 30995 hybrid_clock.cc:648] HybridClock initialized: now 1790563942481299 us; error 56 us; skew 500 ppm
I20260928 02:52:22.481748 30995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:22 dist-test-slave-s1kg krb5kdc[30982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563942, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:22.484965 30995 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:22.486382 30995 webserver.cc:460] Webserver started at http://127.27.229.254:41069/ using document root <none> and password file <none>
I20260928 02:52:22.487123 30995 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:22.487224 30995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:22.487479 30995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:22.490412 30995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "a4cd4d9d89e441e784b55019839e4a5e"
format_stamp: "Formatted at 2026-09-28 02:52:22 on dist-test-slave-s1kg"
server_key: "179c9c101440b0e6b8f07ad21f3bd997"
server_key_iv: "5ce16359611c7d45d40376dcfb648174"
server_key_version: "encryptionkey@0"
I20260928 02:52:22.491211 30995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "a4cd4d9d89e441e784b55019839e4a5e"
format_stamp: "Formatted at 2026-09-28 02:52:22 on dist-test-slave-s1kg"
server_key: "179c9c101440b0e6b8f07ad21f3bd997"
server_key_iv: "5ce16359611c7d45d40376dcfb648174"
server_key_version: "encryptionkey@0"
I20260928 02:52:22.496454 30995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:52:22.499905 31010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:22.501291 30995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:22.501489 30995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "a4cd4d9d89e441e784b55019839e4a5e"
format_stamp: "Formatted at 2026-09-28 02:52:22 on dist-test-slave-s1kg"
server_key: "179c9c101440b0e6b8f07ad21f3bd997"
server_key_iv: "5ce16359611c7d45d40376dcfb648174"
server_key_version: "encryptionkey@0"
I20260928 02:52:22.501704 30995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:22.516343 30995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:22.520906 30995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:22.521217 30995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:22.540302 30995 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34803
I20260928 02:52:22.540323 31061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34803 every 8 connection(s)
I20260928 02:52:22.541725 30995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:22.544358 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 30995
I20260928 02:52:22.544612 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:22.545218 28567 external_mini_cluster.cc:1428] Setting key 3db6b63a3e6a9acc92da50f83511f3bd
I20260928 02:52:22.545678 31062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:22.560148 31062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e: Bootstrap starting.
Sep 28 02:52:22 dist-test-slave-s1kg krb5kdc[30982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563942, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:22.564215 31062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:22.565626 31062 log.cc:824] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:22.569407 31062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e: No bootstrap required, opened a new log
I20260928 02:52:22.571254 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:22.547441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32848 (local address 127.27.229.254:34803)
0928 02:52:22.548305 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:22.548320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:22.548369 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:52:22.550676 (+ 2307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:22.550707 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:22.551307 (+ 600us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:22.551799 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:22.553143 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:22.555087 (+ 1944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:22.556410 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:22.557545 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:22.557753 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:22.562055 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:22.562070 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:22.562093 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:22.562149 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:22.566134 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:22.567215 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:22.567225 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:22.567234 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:22.567432 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:22.567804 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:22.567809 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:22.567812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:22.568552 (+ 740us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:22.568844 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:22.569381 (+ 537us) server_negotiation.cc:299] Negotiation successful
0928 02:52:22.569831 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":364,"threads_started":1}
I20260928 02:52:22.578289 31062 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } }
I20260928 02:52:22.578663 31062 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:22.578795 31062 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cd4d9d89e441e784b55019839e4a5e, State: Initialized, Role: FOLLOWER
I20260928 02:52:22.579396 31062 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } }
I20260928 02:52:22.579592 31062 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:22.579692 31062 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:22.579849 31062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:22.582449 31062 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } }
I20260928 02:52:22.583035 31062 leader_election.cc:302] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cd4d9d89e441e784b55019839e4a5e; no voters:
I20260928 02:52:22.583608 31062 leader_election.cc:288] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:22.583820 31067 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:22.584508 31067 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [term 1 LEADER]: Becoming Leader. State: Replica: a4cd4d9d89e441e784b55019839e4a5e, State: Running, Role: LEADER
I20260928 02:52:22.585034 31067 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } }
I20260928 02:52:22.585711 31062 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:22.588133 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cd4d9d89e441e784b55019839e4a5e. Latest consensus state: current_term: 1 leader_uuid: "a4cd4d9d89e441e784b55019839e4a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } } }
I20260928 02:52:22.588587 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:22.588913 31068 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4cd4d9d89e441e784b55019839e4a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cd4d9d89e441e784b55019839e4a5e" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34803 } } }
I20260928 02:52:22.589234 31068 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:22.590788 31074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:22.595872 31074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:22.605288 31074 catalog_manager.cc:1347] Generated new cluster ID: 31e7a3b6c37746c2a0b658e7ece10426
I20260928 02:52:22.605424 31074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:22.628751 31074 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:22.630746 31074 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:22.643535 31074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4cd4d9d89e441e784b55019839e4a5e: Generated new TSK 0
I20260928 02:52:22.644475 31074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:22.699692 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:23.034193 31088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:23.034775 31088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:23.035007 31088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:23.042497 31088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:23.042637 31088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:23.042758 31088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:23.052736 31088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:23.053998 31088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:23.055294 31088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:23.068935 31096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:23.068960 31094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:23.068996 31093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:23.069927 31088 server_base.cc:1034] running on GCE node
I20260928 02:52:23.070588 31088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:23.071583 31088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:23.072822 31088 hybrid_clock.cc:648] HybridClock initialized: now 1790563943072753 us; error 58 us; skew 500 ppm
I20260928 02:52:23.073143 31088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:23 dist-test-slave-s1kg krb5kdc[30982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:23.076573 31088 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:23.078341 31088 webserver.cc:460] Webserver started at http://127.27.229.193:36525/ using document root <none> and password file <none>
I20260928 02:52:23.079147 31088 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:23.079295 31088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:23.079610 31088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:23.082645 31088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "cb6c3c909127402d99adb4f97fd76412"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "f5a26d25095543cb0da3ea37015bcc6e"
server_key_iv: "24323aa429727b4d9b5dcd46f348717c"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.083547 31088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "cb6c3c909127402d99adb4f97fd76412"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "f5a26d25095543cb0da3ea37015bcc6e"
server_key_iv: "24323aa429727b4d9b5dcd46f348717c"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.089164 31088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:23.093163 31103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.094772 31088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:23.095067 31088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "cb6c3c909127402d99adb4f97fd76412"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "f5a26d25095543cb0da3ea37015bcc6e"
server_key_iv: "24323aa429727b4d9b5dcd46f348717c"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.095287 31088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:23.110443 31088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:23.115149 31088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:23.115464 31088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:23.116493 31088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:23.117913 31088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:23.118019 31088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.118167 31088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:23.118243 31088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.153394 31088 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:43609
I20260928 02:52:23.153476 31215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:43609 every 8 connection(s)
I20260928 02:52:23.154698 31088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:23.159404 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31088
I20260928 02:52:23.159624 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:23.160223 28567 external_mini_cluster.cc:1428] Setting key df88470f237f69e12789c01d2b71e644
Sep 28 02:52:23 dist-test-slave-s1kg krb5kdc[30982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:23.177547 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:23.157478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:56603 (local address 127.27.229.254:34803)
0928 02:52:23.157953 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:23.157958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:23.158824 (+ 866us) server_negotiation.cc:407] Connection header received
0928 02:52:23.160938 (+ 2114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:23.160979 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:23.161071 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:23.161202 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:23.163093 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.164336 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.166035 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.166765 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.166953 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:23.171497 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:23.171502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:23.171506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:23.171542 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:23.174198 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:23.175189 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:23.175193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:23.175195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:23.175314 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:23.175790 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:23.175794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:23.175796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:23.176053 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:23.176208 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:23.176946 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 02:52:23.177176 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":267,"threads_started":1}
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:52:23.178574 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:23.157929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34803 (local address 127.27.229.193:56603)
0928 02:52:23.158652 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:23.158699 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:23.160584 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:23.161477 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:23.161486 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:23.162205 (+ 719us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:23.162913 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:23.162936 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.164518 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.164522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:23.165870 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:23.165884 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.167038 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.167044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:23.167255 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:23.168127 (+ 872us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:23.168164 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:23.171228 (+ 3064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:23.174391 (+ 3163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:23.174401 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:23.174424 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:23.175045 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:23.175434 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:23.175439 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:23.175443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:23.175649 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:23.176188 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:23.176196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:23.176567 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 02:52:23.176836 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 02:52:23.177406 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":288,"threads_started":1}
I20260928 02:52:23.180275 31216 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34803
I20260928 02:52:23.180558 31216 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:23.181326 31216 heartbeater.cc:511] Master 127.27.229.254:34803 requested a full tablet report, sending...
I20260928 02:52:23.183800 31027 ts_manager.cc:194] Registered new tserver with Master: cb6c3c909127402d99adb4f97fd76412 (127.27.229.193:43609)
I20260928 02:52:23.186075 31027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:56603
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:23.207584 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:23.548055 31222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:23.548346 31222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:23.548516 31222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:23.557013 31222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:23.557154 31222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:23.557394 31222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:23.568356 31222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:23.569743 31222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:23.571190 31222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:23.585461 31227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:23.585448 31228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:23.585554 31230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:23.586807 31222 server_base.cc:1034] running on GCE node
I20260928 02:52:23.587594 31222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:23.588675 31222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:23.590032 31222 hybrid_clock.cc:648] HybridClock initialized: now 1790563943589974 us; error 44 us; skew 500 ppm
I20260928 02:52:23.590423 31222 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:23 dist-test-slave-s1kg krb5kdc[30982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:23.594077 31222 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:23.595904 31222 webserver.cc:460] Webserver started at http://127.27.229.194:45805/ using document root <none> and password file <none>
I20260928 02:52:23.596681 31222 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:23.596791 31222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:23.597039 31222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:23.600133 31222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "9ccbf9e287014e31989053f7cc7d4f80"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "41499ef099eb89d9a0cff7d4e32079d7"
server_key_iv: "146769a11957071d0b844ef4b57c355a"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.600960 31222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "9ccbf9e287014e31989053f7cc7d4f80"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "41499ef099eb89d9a0cff7d4e32079d7"
server_key_iv: "146769a11957071d0b844ef4b57c355a"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.606932 31222 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260928 02:52:23.610690 31237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.612332 31222 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:23.612531 31222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "9ccbf9e287014e31989053f7cc7d4f80"
format_stamp: "Formatted at 2026-09-28 02:52:23 on dist-test-slave-s1kg"
server_key: "41499ef099eb89d9a0cff7d4e32079d7"
server_key_iv: "146769a11957071d0b844ef4b57c355a"
server_key_version: "encryptionkey@0"
I20260928 02:52:23.612743 31222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:23.635010 31222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:23.639617 31222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:23.639942 31222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:23.640990 31222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:23.642373 31222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:23.642448 31222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.642598 31222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:23.642671 31222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:23.676793 31222 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:38537
I20260928 02:52:23.676882 31349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:38537 every 8 connection(s)
I20260928 02:52:23.678140 31222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:23.688194 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31222
I20260928 02:52:23.688397 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:23.688819 28567 external_mini_cluster.cc:1428] Setting key 6b63b4dab3c1a3f38ae5ddfec90a53fd
Sep 28 02:52:23 dist-test-slave-s1kg krb5kdc[30982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563943, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:23.700894 31352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:23.680967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37003 (local address 127.27.229.254:34803)
0928 02:52:23.681430 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:23.681438 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:23.682409 (+ 971us) server_negotiation.cc:407] Connection header received
0928 02:52:23.684412 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:23.684416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:23.684487 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:23.684588 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:23.686303 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.687142 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.688918 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.689505 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.689638 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:23.693501 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:23.693510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:23.693518 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:23.693566 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:23.697175 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:23.698311 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:23.698315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:23.698318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:23.698443 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:23.698971 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:23.698975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:23.698977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:23.699261 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:23.699430 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:23.700355 (+ 925us) server_negotiation.cc:299] Negotiation successful
0928 02:52:23.700596 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":228,"threads_started":1}
I20260928 02:52:23.702442 31353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:23.681502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34803 (local address 127.27.229.194:37003)
0928 02:52:23.682239 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:23.682277 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:23.684129 (+ 1852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:23.684869 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:23.684878 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:23.685491 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:23.686091 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:23.686110 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.687308 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.687312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:23.688756 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:23.688769 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:23.689739 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:23.689745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:23.690029 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:23.690927 (+ 898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:23.690962 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:23.693236 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:23.697376 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:23.697387 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:23.697411 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:23.698167 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:23.698556 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:23.698564 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:23.698571 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:23.698858 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:23.699392 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:23.699404 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:23.699914 (+ 510us) client_negotiation.cc:769] Sending connection context
0928 02:52:23.700252 (+ 338us) client_negotiation.cc:240] Negotiation successful
0928 02:52:23.700863 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":287,"threads_started":1}
I20260928 02:52:23.704317 31350 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34803
I20260928 02:52:23.704648 31350 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:23.705474 31350 heartbeater.cc:511] Master 127.27.229.254:34803 requested a full tablet report, sending...
I20260928 02:52:23.707260 31027 ts_manager.cc:194] Registered new tserver with Master: 9ccbf9e287014e31989053f7cc7d4f80 (127.27.229.194:38537)
I20260928 02:52:23.708498 31027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:37003
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:23.736318 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:24.068205 31356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:24.068557 31356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:24.068732 31356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:24.076480 31356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:24.076638 31356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:24.076821 31356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:24.087246 31356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34803
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:24.088689 31356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:24.090018 31356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:24.104410 31364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:24.104465 31362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:24.104373 31361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:24.105868 31356 server_base.cc:1034] running on GCE node
I20260928 02:52:24.106599 31356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:24.107699 31356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:24.108960 31356 hybrid_clock.cc:648] HybridClock initialized: now 1790563944108906 us; error 47 us; skew 500 ppm
I20260928 02:52:24.109318 31356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:24 dist-test-slave-s1kg krb5kdc[30982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:24.112833 31356 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:24.114388 31356 webserver.cc:460] Webserver started at http://127.27.229.195:46121/ using document root <none> and password file <none>
I20260928 02:52:24.115209 31356 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:24.115314 31356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:24.115566 31356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:24.119117 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "a4c620440ef44c0098a7f65d844f671d"
format_stamp: "Formatted at 2026-09-28 02:52:24 on dist-test-slave-s1kg"
server_key: "6ffe9ff8abd3de8f58aefab40c77bf2a"
server_key_iv: "c2c044109cc8f267c951bda8be4eba01"
server_key_version: "encryptionkey@0"
I20260928 02:52:24.119894 31356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "a4c620440ef44c0098a7f65d844f671d"
format_stamp: "Formatted at 2026-09-28 02:52:24 on dist-test-slave-s1kg"
server_key: "6ffe9ff8abd3de8f58aefab40c77bf2a"
server_key_iv: "c2c044109cc8f267c951bda8be4eba01"
server_key_version: "encryptionkey@0"
I20260928 02:52:24.125316 31356 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:24.129014 31371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:24.130478 31356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:52:24.130622 31356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "a4c620440ef44c0098a7f65d844f671d"
format_stamp: "Formatted at 2026-09-28 02:52:24 on dist-test-slave-s1kg"
server_key: "6ffe9ff8abd3de8f58aefab40c77bf2a"
server_key_iv: "c2c044109cc8f267c951bda8be4eba01"
server_key_version: "encryptionkey@0"
I20260928 02:52:24.130806 31356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:24.158061 31356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:24.162647 31356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:24.162993 31356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:24.163949 31356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:24.165292 31356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:24.165369 31356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:24.165453 31356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:24.165498 31356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:24.190428 31216 heartbeater.cc:503] Master 127.27.229.254:34803 was elected leader, sending a full tablet report...
I20260928 02:52:24.199986 31356 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:34699
I20260928 02:52:24.200076 31483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:34699 every 8 connection(s)
I20260928 02:52:24.201221 31356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:24.205817 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31356
I20260928 02:52:24.206007 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:24.206427 28567 external_mini_cluster.cc:1428] Setting key 45d4b5d281f9f4a57284d09e265d9500
Sep 28 02:52:24 dist-test-slave-s1kg krb5kdc[30982](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563944, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:24.221720 31486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:24.203906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:55015 (local address 127.27.229.254:34803)
0928 02:52:24.204368 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:24.204373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:24.205342 (+ 969us) server_negotiation.cc:407] Connection header received
0928 02:52:24.207597 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:24.207602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:24.207698 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:24.207817 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:24.209266 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:24.210017 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:24.211605 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:24.212001 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:24.212139 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:24.215638 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:24.215643 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:24.215646 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:24.215681 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:24.218508 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:24.219339 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:24.219345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:24.219347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:24.219502 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:24.219940 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:24.219947 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:24.219954 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:24.220228 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:24.220388 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:24.221126 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 02:52:24.221352 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260928 02:52:24.222651 31487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:24.204444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34803 (local address 127.27.229.195:55015)
0928 02:52:24.205166 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:24.205209 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:24.207332 (+ 2123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:24.208054 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:24.208062 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:24.208546 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:24.209100 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:24.209117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:24.210168 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:24.210179 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:24.211458 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:24.211473 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:24.212225 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:24.212232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:24.212460 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:24.213155 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:24.213179 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:24.215414 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:24.218660 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:24.218669 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:24.218688 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:24.219200 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:24.219612 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:24.219619 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:24.219624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:24.219820 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:24.220358 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:24.220367 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:24.220732 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 02:52:24.221005 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:52:24.221536 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260928 02:52:24.224118 31484 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34803
I20260928 02:52:24.224469 31484 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:24.225183 31484 heartbeater.cc:511] Master 127.27.229.254:34803 requested a full tablet report, sending...
I20260928 02:52:24.226636 31027 ts_manager.cc:194] Registered new tserver with Master: a4c620440ef44c0098a7f65d844f671d (127.27.229.195:34699)
I20260928 02:52:24.227511 31027 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:55015
I20260928 02:52:24.234016 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:52:24.265168 31486 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:24.262578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.27.229.254:34803)
0928 02:52:24.262900 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:24.262909 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:24.263275 (+ 366us) server_negotiation.cc:407] Connection header received
0928 02:52:24.263431 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:24.263435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:24.263507 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:24.263629 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:24.264982 (+ 1353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32852: BlockingRecv error: recv got EOF from 127.0.0.1:32852 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
I20260928 02:52:24.268492 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31088
W20260928 02:52:24.271276 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31088: Not authorized: Client connection negotiation failed: client connection to 127.27.229.193:43609: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:52:24.271464 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31088
I20260928 02:52:24.287776 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31222
W20260928 02:52:24.290472 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31222: Not authorized: Client connection negotiation failed: client connection to 127.27.229.194:38537: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:52:24.290673 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31222
I20260928 02:52:24.306460 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31356
W20260928 02:52:24.309293 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31356: Not authorized: Client connection negotiation failed: client connection to 127.27.229.195:34699: server requires authentication, but client does not have Kerberos credentials available
I20260928 02:52:24.309413 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31356
I20260928 02:52:24.325759 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 30995
I20260928 02:52:24.327098 31057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:24.438755 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 30995
2026-09-28T02:52:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3633 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:24 dist-test-slave-s1kg krb5kdc[31502](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:24 dist-test-slave-s1kg krb5kdc[31502](info): set up 2 sockets
Sep 28 02:52:24 dist-test-slave-s1kg krb5kdc[31502](info): commencing operation
krb5kdc: starting...
W20260928 02:52:25.531334 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:25 dist-test-slave-s1kg krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:25.732605 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45745
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:45745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:26.060787 31515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:26.061177 31515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:26.061357 31515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:26.068626 31515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:26.068790 31515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:26.068883 31515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:26.068993 31515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:26.069069 31515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:26.079203 31515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:45745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:26.080565 31515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:26.081912 31515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:26.094120 31520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:26.094161 31523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:26.094089 31521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:26.095182 31515 server_base.cc:1034] running on GCE node
I20260928 02:52:26.096071 31515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:26.097771 31515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:26.099014 31515 hybrid_clock.cc:648] HybridClock initialized: now 1790563946098979 us; error 65 us; skew 500 ppm
I20260928 02:52:26.099406 31515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:26 dist-test-slave-s1kg krb5kdc[31502](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563946, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:26.106686 31515 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:26.108373 31515 webserver.cc:460] Webserver started at http://127.27.229.254:38033/ using document root <none> and password file <none>
I20260928 02:52:26.109140 31515 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:26.109297 31515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:26.109616 31515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:26.112743 31515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "fc07c5813c8445729f2eb4fdbb0d979a"
format_stamp: "Formatted at 2026-09-28 02:52:26 on dist-test-slave-s1kg"
server_key: "c1a3bfc620bf294f22de28a0e1edc464"
server_key_iv: "db3ed24890ba6173bae0b0bbab50f354"
server_key_version: "encryptionkey@0"
I20260928 02:52:26.113583 31515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "fc07c5813c8445729f2eb4fdbb0d979a"
format_stamp: "Formatted at 2026-09-28 02:52:26 on dist-test-slave-s1kg"
server_key: "c1a3bfc620bf294f22de28a0e1edc464"
server_key_iv: "db3ed24890ba6173bae0b0bbab50f354"
server_key_version: "encryptionkey@0"
I20260928 02:52:26.118767 31515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:52:26.122295 31530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:26.123910 31515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:26.124112 31515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "fc07c5813c8445729f2eb4fdbb0d979a"
format_stamp: "Formatted at 2026-09-28 02:52:26 on dist-test-slave-s1kg"
server_key: "c1a3bfc620bf294f22de28a0e1edc464"
server_key_iv: "db3ed24890ba6173bae0b0bbab50f354"
server_key_version: "encryptionkey@0"
I20260928 02:52:26.124316 31515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:26.166172 31515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:26.170725 31515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:26.171059 31515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:26.189978 31515 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:45745
I20260928 02:52:26.190070 31581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:45745 every 8 connection(s)
I20260928 02:52:26.191433 31515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:26.192620 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31515
I20260928 02:52:26.192867 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:26.193688 28567 external_mini_cluster.cc:1428] Setting key eb8995ec0a95036508f4028acbc7ee4e
I20260928 02:52:26.196674 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:52:26 dist-test-slave-s1kg krb5kdc[31502](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563945, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:26.210292 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a: Bootstrap starting.
I20260928 02:52:26.214496 31582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:26.215828 31582 log.cc:824] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:26.219445 31582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a: No bootstrap required, opened a new log
I20260928 02:52:26.220219 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:26.195934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.27.229.254:45745)
0928 02:52:26.196746 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:26.196761 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:26.196811 (+ 50us) server_negotiation.cc:407] Connection header received
0928 02:52:26.199116 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:26.199148 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:26.199693 (+ 545us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:26.200261 (+ 568us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:26.201793 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:26.203953 (+ 2160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:26.205327 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:26.206402 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:26.206605 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:26.211015 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:26.211027 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:26.211047 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:26.211099 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:26.215165 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:26.216062 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:26.216071 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:26.216080 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:26.216295 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:26.216845 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:26.216850 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:26.216853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:26.217585 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:26.217884 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:26.218367 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 02:52:26.218805 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":337,"threads_started":1}
I20260928 02:52:26.224711 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:52:26.230513 31582 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } }
I20260928 02:52:26.231103 31582 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:26.231218 31582 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc07c5813c8445729f2eb4fdbb0d979a, State: Initialized, Role: FOLLOWER
I20260928 02:52:26.231909 31582 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } }
I20260928 02:52:26.232224 31582 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:26.232369 31582 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:26.232565 31582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:26.234782 31582 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } }
I20260928 02:52:26.235409 31582 leader_election.cc:302] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc07c5813c8445729f2eb4fdbb0d979a; no voters:
I20260928 02:52:26.236097 31582 leader_election.cc:288] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:26.236222 31588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:26.238849 31582 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:26.239266 31588 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [term 1 LEADER]: Becoming Leader. State: Replica: fc07c5813c8445729f2eb4fdbb0d979a, State: Running, Role: LEADER
I20260928 02:52:26.239924 31588 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } }
I20260928 02:52:26.243516 31589 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } } }
I20260928 02:52:26.244006 31589 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:26.243474 31590 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc07c5813c8445729f2eb4fdbb0d979a. Latest consensus state: current_term: 1 leader_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc07c5813c8445729f2eb4fdbb0d979a" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45745 } } }
I20260928 02:52:26.244299 31590 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:26.250697 31599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:26.256947 31599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:26.268652 31599 catalog_manager.cc:1347] Generated new cluster ID: f5bc612de8e649a89d91f72d0f5da3c2
I20260928 02:52:26.268901 31599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:26.279052 31585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:26.267221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.27.229.254:45745)
0928 02:52:26.267404 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:26.267410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:26.269460 (+ 2050us) server_negotiation.cc:407] Connection header received
0928 02:52:26.269677 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:26.269683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:26.269799 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:26.269918 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:26.271309 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:26.272432 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:26.274436 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:26.275394 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:26.275568 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:26.276982 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:26.276990 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:26.276997 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:26.278044 (+ 1047us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:26.278122 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:26.278154 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:26.278286 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:26.278449 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 02:52:26.278556 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:52:26.288228 31599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:26.290174 31599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:26.308357 31599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc07c5813c8445729f2eb4fdbb0d979a: Generated new TSK 0
I20260928 02:52:26.309412 31599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:52:26.325951 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31515
I20260928 02:52:26.327467 31577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:26.420329 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31515
2026-09-28T02:52:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1975 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:26 dist-test-slave-s1kg krb5kdc[31616](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:26 dist-test-slave-s1kg krb5kdc[31616](info): set up 2 sockets
Sep 28 02:52:26 dist-test-slave-s1kg krb5kdc[31616](info): commencing operation
krb5kdc: starting...
W20260928 02:52:27.511469 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:27 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:27.714702 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33265
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:33265
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:28.060782 31629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:28.061156 31629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:28.061347 31629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:28.069963 31629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:28.070132 31629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:28.070220 31629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:28.070309 31629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:28.070395 31629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:28.080494 31629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:33265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:28.081964 31629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:28.083374 31629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:28.095367 31634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:28.095383 31635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:28.095650 31637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:28.096237 31629 server_base.cc:1034] running on GCE node
I20260928 02:52:28.097008 31629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:28.098399 31629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:28.099597 31629 hybrid_clock.cc:648] HybridClock initialized: now 1790563948099540 us; error 49 us; skew 500 ppm
I20260928 02:52:28.099895 31629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:28 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563948, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:28.103137 31629 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:28.104768 31629 webserver.cc:460] Webserver started at http://127.27.229.254:38567/ using document root <none> and password file <none>
I20260928 02:52:28.105480 31629 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:28.105628 31629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:28.105945 31629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:28.109066 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "e28262dcca9b4ed7b3a9ba4701608add"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "c1d1d2dbf602e4a23a3a91b9a3249321"
server_key_iv: "511a8277b219f45b41194730a61cb33b"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.109901 31629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "e28262dcca9b4ed7b3a9ba4701608add"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "c1d1d2dbf602e4a23a3a91b9a3249321"
server_key_iv: "511a8277b219f45b41194730a61cb33b"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.115419 31629 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:28.119267 31644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:28.120811 31629 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:28.121047 31629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "e28262dcca9b4ed7b3a9ba4701608add"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "c1d1d2dbf602e4a23a3a91b9a3249321"
server_key_iv: "511a8277b219f45b41194730a61cb33b"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.121268 31629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:28.143661 31629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:28.148504 31629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:28.148844 31629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:28.168552 31629 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:33265
I20260928 02:52:28.168605 31695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:33265 every 8 connection(s)
I20260928 02:52:28.169966 31629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:28.173791 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:28.174476 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31629
I20260928 02:52:28.174886 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:28.175531 28567 external_mini_cluster.cc:1428] Setting key ebfbf8f1dc28ce881010bb93890eb90b
I20260928 02:52:28.188552 31696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add: Bootstrap starting.
Sep 28 02:52:28 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:28.192377 31696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:28.193620 31696 log.cc:824] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:28.196693 31696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add: No bootstrap required, opened a new log
I20260928 02:52:28.199630 31699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:28.177727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59566 (local address 127.27.229.254:33265)
0928 02:52:28.178596 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:28.178608 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:28.178653 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:52:28.180326 (+ 1673us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:28.180350 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:28.180788 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:28.181159 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:28.182991 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.184521 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.186373 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.187206 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.187353 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:28.192079 (+ 4726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:28.192088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:28.192104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:28.192148 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:28.195345 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:28.196088 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:28.196095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:28.196101 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:28.196260 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:28.196642 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:28.196647 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:28.196650 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:28.197233 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:28.197618 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:28.198095 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 02:52:28.198533 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":415,"threads_started":1}
I20260928 02:52:28.204922 31696 raft_consensus.cc:348] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } }
I20260928 02:52:28.205440 31696 raft_consensus.cc:374] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:28.205574 31696 raft_consensus.cc:729] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e28262dcca9b4ed7b3a9ba4701608add, State: Initialized, Role: FOLLOWER
I20260928 02:52:28.206241 31696 consensus_queue.cc:260] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } }
I20260928 02:52:28.206453 31696 raft_consensus.cc:388] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:28.206557 31696 raft_consensus.cc:482] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:28.206672 31696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:28.208868 31696 raft_consensus.cc:504] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } }
I20260928 02:52:28.209383 31696 leader_election.cc:302] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e28262dcca9b4ed7b3a9ba4701608add; no voters:
I20260928 02:52:28.210153 31696 leader_election.cc:288] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:28.210268 31701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:28.211127 31701 raft_consensus.cc:686] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [term 1 LEADER]: Becoming Leader. State: Replica: e28262dcca9b4ed7b3a9ba4701608add, State: Running, Role: LEADER
I20260928 02:52:28.211731 31701 consensus_queue.cc:237] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } }
I20260928 02:52:28.212455 31696 sys_catalog.cc:564] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:28.214494 31702 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e28262dcca9b4ed7b3a9ba4701608add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } } }
I20260928 02:52:28.215037 31702 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:28.214573 31703 sys_catalog.cc:455] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [sys.catalog]: SysCatalogTable state changed. Reason: New leader e28262dcca9b4ed7b3a9ba4701608add. Latest consensus state: current_term: 1 leader_uuid: "e28262dcca9b4ed7b3a9ba4701608add" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e28262dcca9b4ed7b3a9ba4701608add" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33265 } } }
I20260928 02:52:28.215214 31703 sys_catalog.cc:458] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:28.217559 31707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:28.222710 31707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:28.233224 31707 catalog_manager.cc:1347] Generated new cluster ID: 138fb47c8b164eb8aa36d798656ac2ec
I20260928 02:52:28.233393 31707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:28.247534 31707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:28.248881 31707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:28.259404 31707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e28262dcca9b4ed7b3a9ba4701608add: Generated new TSK 0
I20260928 02:52:28.260548 31707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:28.318409 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:28.647593 31722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:28.647897 31722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:28.648077 31722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:28.655424 31722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:28.655543 31722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:28.655748 31722 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:28.665488 31722 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:28.666899 31722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:28.668201 31722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:28.681164 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:28.681226 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:28.681258 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:28.682457 31722 server_base.cc:1034] running on GCE node
I20260928 02:52:28.683274 31722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:28.684284 31722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:28.685550 31722 hybrid_clock.cc:648] HybridClock initialized: now 1790563948685493 us; error 49 us; skew 500 ppm
I20260928 02:52:28.685945 31722 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:28 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563948, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:28.689530 31722 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:28.691191 31722 webserver.cc:460] Webserver started at http://127.27.229.193:43433/ using document root <none> and password file <none>
I20260928 02:52:28.691888 31722 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:28.692019 31722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:28.692348 31722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:28.695384 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "287bbc3406e543e6a6ac0c5f032c7a14"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "43d0cffc4c7c21be66110031bd61c1c7"
server_key_iv: "c6c362228c9ccaec1b8a48209d9e1a57"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.696177 31722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "287bbc3406e543e6a6ac0c5f032c7a14"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "43d0cffc4c7c21be66110031bd61c1c7"
server_key_iv: "c6c362228c9ccaec1b8a48209d9e1a57"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.701187 31722 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:28.704948 31737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:28.706411 31722 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:28.706594 31722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "287bbc3406e543e6a6ac0c5f032c7a14"
format_stamp: "Formatted at 2026-09-28 02:52:28 on dist-test-slave-s1kg"
server_key: "43d0cffc4c7c21be66110031bd61c1c7"
server_key_iv: "c6c362228c9ccaec1b8a48209d9e1a57"
server_key_version: "encryptionkey@0"
I20260928 02:52:28.706807 31722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:28.731972 31722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:28.736714 31722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:28.737033 31722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:28.738025 31722 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:28.739625 31722 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:28.739728 31722 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:28.739873 31722 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:28.739948 31722 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:28.773352 31722 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:36193
I20260928 02:52:28.773442 31849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:36193 every 8 connection(s)
I20260928 02:52:28.774631 31722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:28.779613 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31722
I20260928 02:52:28.779840 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:28.780473 28567 external_mini_cluster.cc:1428] Setting key 69fae5d666560b944c3b2a1b974bebed
Sep 28 02:52:28 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563948, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:28.797183 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:28.777266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48631 (local address 127.27.229.254:33265)
0928 02:52:28.777792 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:28.777799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:28.778771 (+ 972us) server_negotiation.cc:407] Connection header received
0928 02:52:28.780746 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:28.780751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:28.780838 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:28.780970 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:28.782754 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.783913 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.785613 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.786348 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.786500 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:28.790242 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:28.790252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:28.790260 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:28.790304 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:28.793767 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:28.794503 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:28.794511 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:28.794518 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:28.794664 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:28.795151 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:28.795159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:28.795165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:28.795539 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:28.795700 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:28.796447 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 02:52:28.796721 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":236,"threads_started":1}
I20260928 02:52:28.797942 31853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:28.777918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33265 (local address 127.27.229.193:48631)
0928 02:52:28.778609 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:28.778645 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:28.780428 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:28.781281 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:28.781295 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:28.781856 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:28.782563 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:28.782584 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.784093 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.784097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:28.785455 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:28.785467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:28.786568 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:28.786573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:28.786805 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:28.787676 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:28.787714 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:28.790017 (+ 2303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:28.793939 (+ 3922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:28.793946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:28.793961 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:28.794380 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:28.794783 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:28.794793 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:28.794796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:28.795041 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:28.795666 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:28.795673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:28.796014 (+ 341us) client_negotiation.cc:769] Sending connection context
0928 02:52:28.796275 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 02:52:28.796780 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":271,"threads_started":1}
I20260928 02:52:28.799595 31850 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33265
I20260928 02:52:28.799916 31850 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:52:28.800623 31850 heartbeater.cc:511] Master 127.27.229.254:33265 requested a full tablet report, sending...
I20260928 02:52:28.802812 31661 ts_manager.cc:194] Registered new tserver with Master: 287bbc3406e543e6a6ac0c5f032c7a14 (127.27.229.193:36193)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:28.804386 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:48631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:28.827051 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:29.153175 31856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:29.153517 31856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:29.153693 31856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:29.161188 31856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:29.161356 31856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:29.161540 31856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:29.171566 31856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:29.172848 31856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:29.174150 31856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:29.187975 31864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:29.188023 31861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:29.187996 31862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:29.188617 31856 server_base.cc:1034] running on GCE node
I20260928 02:52:29.189261 31856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:29.190217 31856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:29.191471 31856 hybrid_clock.cc:648] HybridClock initialized: now 1790563949191422 us; error 61 us; skew 500 ppm
I20260928 02:52:29.191881 31856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:29.195267 31856 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:29.197073 31856 webserver.cc:460] Webserver started at http://127.27.229.194:34433/ using document root <none> and password file <none>
I20260928 02:52:29.197853 31856 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:29.198017 31856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:29.198349 31856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:29.201602 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "d1634a50b6d049daa84d3eb8eb890592"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "079b24737754e7f86c204ae03b6bcc1b"
server_key_iv: "b0be22fe150d73632768176e1c8e3ddb"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.202478 31856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "d1634a50b6d049daa84d3eb8eb890592"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "079b24737754e7f86c204ae03b6bcc1b"
server_key_iv: "b0be22fe150d73632768176e1c8e3ddb"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.208333 31856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:52:29.212335 31871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.213922 31856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:29.214111 31856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "d1634a50b6d049daa84d3eb8eb890592"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "079b24737754e7f86c204ae03b6bcc1b"
server_key_iv: "b0be22fe150d73632768176e1c8e3ddb"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.214299 31856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:29.226263 31856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:29.230427 31856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:29.230727 31856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:29.231786 31856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:29.233134 31856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:29.233206 31856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.233379 31856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:29.233430 31856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.269037 31856 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:35171
I20260928 02:52:29.269069 31983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:35171 every 8 connection(s)
I20260928 02:52:29.270311 31856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:29.278271 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31856
I20260928 02:52:29.278481 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:29.279002 28567 external_mini_cluster.cc:1428] Setting key 2db10e595d7ecdd2460a60ca1141e631
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:29.293450 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.273265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:35893 (local address 127.27.229.254:33265)
0928 02:52:29.273540 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.273545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.274687 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 02:52:29.276638 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.276644 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.276758 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.276874 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.278590 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.279443 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.281343 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.281763 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.281915 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.286027 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.286033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.286036 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.286070 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.289574 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.290412 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.290420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.290426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.290583 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.291183 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.291191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.291196 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.291583 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.291748 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.292635 (+ 887us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.293010 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260928 02:52:29.294214 31986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.273740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33265 (local address 127.27.229.194:35893)
0928 02:52:29.274498 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:29.274546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:29.276340 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:29.277138 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:29.277147 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:29.277683 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:29.278326 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:29.278344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.279616 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.279620 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:29.281183 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:29.281196 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.282006 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.282012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:29.282257 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.283064 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:29.283088 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:29.285746 (+ 2658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:29.289770 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:29.289778 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:29.289794 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:29.290269 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:29.290737 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:29.290741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:29.290743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:29.291063 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:29.291749 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:29.291758 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:29.292201 (+ 443us) client_negotiation.cc:769] Sending connection context
0928 02:52:29.292460 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 02:52:29.293008 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":317,"threads_started":1}
I20260928 02:52:29.296088 31984 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33265
I20260928 02:52:29.296510 31984 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:29.297516 31984 heartbeater.cc:511] Master 127.27.229.254:33265 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:52:29.299484 31661 ts_manager.cc:194] Registered new tserver with Master: d1634a50b6d049daa84d3eb8eb890592 (127.27.229.194:35171)
I20260928 02:52:29.300364 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:35893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:29.327255 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:29.653391 31989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:29.653832 31989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:29.654016 31989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:29.661315 31989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:29.661440 31989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:29.661554 31989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:29.671293 31989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33265
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:29.672566 31989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:29.673866 31989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:29.687463 31994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:29.687434 31995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:29.687664 31997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:29.688619 31989 server_base.cc:1034] running on GCE node
I20260928 02:52:29.689306 31989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:29.690356 31989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:29.691569 31989 hybrid_clock.cc:648] HybridClock initialized: now 1790563949691518 us; error 48 us; skew 500 ppm
I20260928 02:52:29.691880 31989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:29.695276 31989 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:29.696962 31989 webserver.cc:460] Webserver started at http://127.27.229.195:44793/ using document root <none> and password file <none>
I20260928 02:52:29.697649 31989 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:29.697753 31989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:29.697997 31989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:29.700942 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "145870320b56488cb19a499d83f351b4"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "1511964073ae041b4adbb74ac0627e54"
server_key_iv: "948339656907929015ed4714f35d44b4"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.701689 31989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "145870320b56488cb19a499d83f351b4"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "1511964073ae041b4adbb74ac0627e54"
server_key_iv: "948339656907929015ed4714f35d44b4"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.707238 31989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:52:29.711406 32004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.713174 31989 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:52:29.713363 31989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "145870320b56488cb19a499d83f351b4"
format_stamp: "Formatted at 2026-09-28 02:52:29 on dist-test-slave-s1kg"
server_key: "1511964073ae041b4adbb74ac0627e54"
server_key_iv: "948339656907929015ed4714f35d44b4"
server_key_version: "encryptionkey@0"
I20260928 02:52:29.713579 31989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:29.732407 31989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:29.736940 31989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:29.737278 31989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:29.738314 31989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:29.739697 31989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:29.739779 31989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.739916 31989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:29.739967 31989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:29.773766 31989 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:38707
I20260928 02:52:29.773795 32116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:38707 every 8 connection(s)
I20260928 02:52:29.775453 31989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:29.778251 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 31989
I20260928 02:52:29.778458 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:29.779049 28567 external_mini_cluster.cc:1428] Setting key 3f3bbc6a59842e3160f19d60ea48547e
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:29.796507 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.778668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:39989 (local address 127.27.229.254:33265)
0928 02:52:29.778934 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.778940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.779865 (+ 925us) server_negotiation.cc:407] Connection header received
0928 02:52:29.782080 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.782086 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.782165 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.782280 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.783979 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.784787 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.786457 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.786971 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.787077 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.790537 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.790542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.790545 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.790576 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.793260 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.794133 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.794138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.794141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.794295 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.794813 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.794818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.794820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.795206 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.795338 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.796053 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.796255 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:52:29.797499 32119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.778756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33265 (local address 127.27.229.195:39989)
0928 02:52:29.779685 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:29.779733 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:29.781764 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:29.782607 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:29.782615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:29.783237 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:29.783794 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:29.783810 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.784955 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.784959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:29.786306 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:29.786318 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.787220 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.787226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:29.787425 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.788059 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:29.788086 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:29.790339 (+ 2253us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:29.793415 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:29.793424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:29.793439 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:29.794005 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:29.794430 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:29.794435 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:29.794438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:29.794697 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:29.795350 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:29.795357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:29.795685 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:52:29.795930 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 02:52:29.796367 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":341,"threads_started":1}
I20260928 02:52:29.798983 32117 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33265
I20260928 02:52:29.799292 32117 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:29.800050 32117 heartbeater.cc:511] Master 127.27.229.254:33265 requested a full tablet report, sending...
I20260928 02:52:29.801518 31661 ts_manager.cc:194] Registered new tserver with Master: 145870320b56488cb19a499d83f351b4 (127.27.229.195:38707)
I20260928 02:52:29.802417 31661 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:39989
I20260928 02:52:29.807366 31850 heartbeater.cc:503] Master 127.27.229.254:33265 was elected leader, sending a full tablet report...
I20260928 02:52:29.807536 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:29.856088 31852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.842257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47190 (local address 127.27.229.254:33265)
0928 02:52:29.842447 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.842452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.842926 (+ 474us) server_negotiation.cc:407] Connection header received
0928 02:52:29.843130 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.843134 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.843234 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.843331 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.844665 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.845414 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.846587 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.847073 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.847171 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.850915 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.850920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.850924 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.850958 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.853563 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.854192 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.854196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.854199 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.854360 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.854690 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.854693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.854696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.855099 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.855303 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.855677 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.855848 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260928 02:52:29.856645 31661 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47190
I20260928 02:52:29.858266 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31722
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:29.877674 32128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.859676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40262 (local address 127.27.229.193:36193)
0928 02:52:29.863313 (+ 3637us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.863321 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.863346 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:52:29.863406 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.863413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.863623 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.863795 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.865033 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.866033 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.867301 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.867847 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.867996 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.871819 (+ 3823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.871831 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.871846 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.871877 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.874582 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.875388 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.875401 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.875404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.875523 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.875933 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.875954 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.875956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.876397 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.876701 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.876986 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.877242 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3490,"thread_start_us":3339,"threads_started":1}
W20260928 02:52:29.879024 31845 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40262
W20260928 02:52:29.880119 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31722: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:29.880306 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31722
I20260928 02:52:29.895980 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31856
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:29.911469 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.896892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39498 (local address 127.27.229.194:35171)
0928 02:52:29.897449 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.897455 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.897482 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:52:29.897540 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.897546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.897755 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.897928 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.899184 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.900214 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.901436 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.901976 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.902070 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.906024 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.906036 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.906051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.906083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.908980 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.909608 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.909615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.909617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.909732 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.910024 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.910028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.910030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.910341 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.910553 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.910804 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.911082 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":304,"threads_started":1}
W20260928 02:52:29.912899 31979 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39498
W20260928 02:52:29.914101 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:29.914258 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31856
I20260928 02:52:29.930269 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31989
Sep 28 02:52:29 dist-test-slave-s1kg krb5kdc[31616](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563949, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:29.947261 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:29.931267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38382 (local address 127.27.229.195:38707)
0928 02:52:29.931936 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:29.931943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:29.931971 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:52:29.932031 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:29.932038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:29.932244 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:29.932420 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:29.933714 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.934736 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.935971 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:29.936563 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:29.936700 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:29.941200 (+ 4500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:29.941213 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:29.941228 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:29.941281 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:29.944411 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.945263 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.945270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.945273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.945406 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:29.945742 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:29.945745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:29.945747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:29.946122 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:29.946351 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:29.946643 (+ 292us) server_negotiation.cc:299] Negotiation successful
0928 02:52:29.946967 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":346,"threads_started":1}
W20260928 02:52:29.948549 32112 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38382
W20260928 02:52:29.949622 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31989: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:52:29.949838 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31989
I20260928 02:52:29.965149 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 31629
I20260928 02:52:29.966408 31691 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:30.069162 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 31629
2026-09-28T02:52:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3654 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:30 dist-test-slave-s1kg krb5kdc[32134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:30 dist-test-slave-s1kg krb5kdc[32134](info): set up 2 sockets
Sep 28 02:52:30 dist-test-slave-s1kg krb5kdc[32134](info): commencing operation
krb5kdc: starting...
W20260928 02:52:31.155171 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:31 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:31.363454 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46367
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:31.689376 32147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:31.689663 32147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:31.689813 32147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:31.697270 32147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:31.697377 32147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:31.697463 32147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:31.697567 32147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:31.697647 32147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:31.707437 32147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:31.708750 32147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:31.710129 32147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:31.722437 32152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:31.722579 32153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:31.722466 32155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:31.723925 32147 server_base.cc:1034] running on GCE node
I20260928 02:52:31.724730 32147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:31.726163 32147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:31.727401 32147 hybrid_clock.cc:648] HybridClock initialized: now 1790563951727371 us; error 58 us; skew 500 ppm
I20260928 02:52:31.727752 32147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:31 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:31.735428 32147 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:31.737046 32147 webserver.cc:460] Webserver started at http://127.27.229.254:36633/ using document root <none> and password file <none>
I20260928 02:52:31.737741 32147 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:31.737843 32147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:31.738101 32147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:31.741142 32147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "3655ff7130cc41df8d89ca5c60196d60"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-s1kg"
server_key: "737bdf606c10f32d2a9e8969c11d49e9"
server_key_iv: "4e16d1f45b904abba06a34a478859a7c"
server_key_version: "encryptionkey@0"
I20260928 02:52:31.741885 32147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "3655ff7130cc41df8d89ca5c60196d60"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-s1kg"
server_key: "737bdf606c10f32d2a9e8969c11d49e9"
server_key_iv: "4e16d1f45b904abba06a34a478859a7c"
server_key_version: "encryptionkey@0"
I20260928 02:52:31.747323 32147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:31.751029 32162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:31.752434 32147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:31.752624 32147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "3655ff7130cc41df8d89ca5c60196d60"
format_stamp: "Formatted at 2026-09-28 02:52:31 on dist-test-slave-s1kg"
server_key: "737bdf606c10f32d2a9e8969c11d49e9"
server_key_iv: "4e16d1f45b904abba06a34a478859a7c"
server_key_version: "encryptionkey@0"
I20260928 02:52:31.752851 32147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:31.769091 32147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:31.773679 32147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:31.773999 32147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:31.792716 32147 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:46367
I20260928 02:52:31.792771 32213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:46367 every 8 connection(s)
I20260928 02:52:31.794080 32147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:31.794667 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32147
I20260928 02:52:31.794986 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:31.795810 28567 external_mini_cluster.cc:1428] Setting key 5951f54a463ad90700b4a343eb3763c3
I20260928 02:52:31.798319 32214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:31.808692 32214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60: Bootstrap starting.
Sep 28 02:52:31 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:31.811834 32214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:31.812925 32214 log.cc:824] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:31.815994 32214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60: No bootstrap required, opened a new log
I20260928 02:52:31.819442 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:31.797801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35860 (local address 127.27.229.254:46367)
0928 02:52:31.798632 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:31.798644 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:31.798685 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:52:31.800493 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:31.800514 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:31.800922 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:31.801358 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:31.802941 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.804365 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.805699 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:31.806461 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:31.806696 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:31.810975 (+ 4279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:31.810983 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:31.811003 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:31.811041 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:31.814954 (+ 3913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.815721 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.815734 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.815741 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.815908 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:31.816381 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:31.816386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:31.816388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:31.816995 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:31.817326 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:31.817827 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 02:52:31.818275 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":332,"threads_started":1}
I20260928 02:52:31.824311 32214 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } }
I20260928 02:52:31.824671 32214 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:31.824771 32214 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3655ff7130cc41df8d89ca5c60196d60, State: Initialized, Role: FOLLOWER
I20260928 02:52:31.825294 32214 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } }
I20260928 02:52:31.825524 32214 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:31.825665 32214 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:31.825826 32214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:31.827957 32214 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } }
I20260928 02:52:31.828481 32214 leader_election.cc:302] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3655ff7130cc41df8d89ca5c60196d60; no voters:
I20260928 02:52:31.829074 32214 leader_election.cc:288] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:31.829242 32219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:31.829962 32219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [term 1 LEADER]: Becoming Leader. State: Replica: 3655ff7130cc41df8d89ca5c60196d60, State: Running, Role: LEADER
I20260928 02:52:31.830503 32219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } }
I20260928 02:52:31.831101 32214 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:31.833425 32220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3655ff7130cc41df8d89ca5c60196d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } } }
I20260928 02:52:31.833906 32220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:31.835543 32221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3655ff7130cc41df8d89ca5c60196d60. Latest consensus state: current_term: 1 leader_uuid: "3655ff7130cc41df8d89ca5c60196d60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3655ff7130cc41df8d89ca5c60196d60" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46367 } } }
I20260928 02:52:31.839277 32221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:31.844141 32227 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:31.848457 32227 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:31.857438 32227 catalog_manager.cc:1347] Generated new cluster ID: fd24159d9c5c4a3f8a8a7774fb283cd5
I20260928 02:52:31.857563 32227 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:31.866030 32227 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:31.867807 32227 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:31.876291 32227 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3655ff7130cc41df8d89ca5c60196d60: Generated new TSK 0
I20260928 02:52:31.877462 32227 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:31.938167 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:32.266499 32240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:32.266875 32240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:32.267048 32240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:32.274333 32240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:32.274503 32240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:32.274700 32240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:32.284751 32240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:32.286222 32240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:32.287604 32240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:32.301138 32245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:32.301203 32246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:32.301157 32248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:32.302039 32240 server_base.cc:1034] running on GCE node
I20260928 02:52:32.302663 32240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:32.303699 32240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:32.304953 32240 hybrid_clock.cc:648] HybridClock initialized: now 1790563952304925 us; error 61 us; skew 500 ppm
I20260928 02:52:32.305294 32240 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:32 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:32.308748 32240 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:32.310228 32240 webserver.cc:460] Webserver started at http://127.27.229.193:38413/ using document root <none> and password file <none>
I20260928 02:52:32.310953 32240 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:32.311060 32240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:32.311306 32240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:32.314093 32240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "14de0991952445709636788f625e7a61"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "a078dcc547fc16950b661514d6639c91"
server_key_iv: "5d95274493bb839035ce401c7a407631"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.314960 32240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "14de0991952445709636788f625e7a61"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "a078dcc547fc16950b661514d6639c91"
server_key_iv: "5d95274493bb839035ce401c7a407631"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.320278 32240 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:52:32.323787 32255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.325098 32240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:52:32.325261 32240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "14de0991952445709636788f625e7a61"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "a078dcc547fc16950b661514d6639c91"
server_key_iv: "5d95274493bb839035ce401c7a407631"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.325457 32240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:32.337682 32240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:32.342183 32240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:32.342475 32240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:32.343502 32240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:32.345043 32240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:32.345127 32240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.345233 32240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:32.345299 32240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.380687 32240 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39001
I20260928 02:52:32.380735 32367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39001 every 8 connection(s)
I20260928 02:52:32.382046 32240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:32.389930 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32240
I20260928 02:52:32.390208 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:32.390923 28567 external_mini_cluster.cc:1428] Setting key 8a52f6ef6dd63cbf214c3f3efc49b6bb
Sep 28 02:52:32 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:32.404199 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:32.385073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:56655 (local address 127.27.229.254:46367)
0928 02:52:32.385552 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:32.385556 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:32.386402 (+ 846us) server_negotiation.cc:407] Connection header received
0928 02:52:32.388271 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:32.388277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:32.388370 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:32.388491 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:32.390040 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.390956 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.393191 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.393585 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.393687 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:32.398039 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:32.398045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:32.398048 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:32.398079 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:32.400705 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:32.401656 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:32.401666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:32.401670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:32.401814 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:32.402297 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:32.402302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:32.402307 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:32.402582 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:32.402758 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:32.403587 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 02:52:32.403832 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":264,"threads_started":1}
I20260928 02:52:32.405227 32371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:32.385494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46367 (local address 127.27.229.193:56655)
0928 02:52:32.386220 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:32.386270 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:32.387991 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:32.388723 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:32.388739 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:32.389249 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:32.389859 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:32.389877 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.391144 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.391151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:32.393021 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:32.393037 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.393841 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.393851 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:32.394187 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:32.395089 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:32.395127 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:32.397807 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:32.400875 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:32.400885 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:32.400905 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:32.401355 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:32.401945 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:32.401951 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:32.401954 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:32.402156 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:32.402731 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:32.402739 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:32.403193 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 02:52:32.403485 (+ 292us) client_negotiation.cc:240] Negotiation successful
0928 02:52:32.404069 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":307,"threads_started":1}
I20260928 02:52:32.407084 32368 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46367
I20260928 02:52:32.407413 32368 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:32.408145 32368 heartbeater.cc:511] Master 127.27.229.254:46367 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:52:32.410522 32179 ts_manager.cc:194] Registered new tserver with Master: 14de0991952445709636788f625e7a61 (127.27.229.193:39001)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:32.412452 32179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:56655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:32.438258 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:32.760376 32374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:32.760696 32374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:32.760905 32374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:32.769187 32374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:32.769302 32374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:32.769487 32374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:32.779608 32374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:32.780964 32374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:32.782322 32374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:32.796046 32379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:32.796473 32380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:32.796717 32382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:32.797995 32374 server_base.cc:1034] running on GCE node
I20260928 02:52:32.798645 32374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:32.799808 32374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:32.801036 32374 hybrid_clock.cc:648] HybridClock initialized: now 1790563952800980 us; error 53 us; skew 500 ppm
I20260928 02:52:32.801362 32374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:32 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:32.804859 32374 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:32.806502 32374 webserver.cc:460] Webserver started at http://127.27.229.194:37675/ using document root <none> and password file <none>
I20260928 02:52:32.807296 32374 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:32.807437 32374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:32.807739 32374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:32.810777 32374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "9553126427cb45fd9ce2df66fdeb425b"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "ad18969d090b4d56b4b53da5dbf8e4d0"
server_key_iv: "c379abd4fee11fdd4125a46c9f36d8a9"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.811605 32374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "9553126427cb45fd9ce2df66fdeb425b"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "ad18969d090b4d56b4b53da5dbf8e4d0"
server_key_iv: "c379abd4fee11fdd4125a46c9f36d8a9"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.816776 32374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:52:32.820447 32389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.822034 32374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:32.822250 32374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "9553126427cb45fd9ce2df66fdeb425b"
format_stamp: "Formatted at 2026-09-28 02:52:32 on dist-test-slave-s1kg"
server_key: "ad18969d090b4d56b4b53da5dbf8e4d0"
server_key_iv: "c379abd4fee11fdd4125a46c9f36d8a9"
server_key_version: "encryptionkey@0"
I20260928 02:52:32.822504 32374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:32.837186 32374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:32.841894 32374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:32.842181 32374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:32.843197 32374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:32.844551 32374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:32.844651 32374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.844811 32374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:32.844866 32374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:32.879081 32374 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:40529
I20260928 02:52:32.879137 32501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:40529 every 8 connection(s)
I20260928 02:52:32.880403 32374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:32.888099 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32374
I20260928 02:52:32.888303 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:32.888830 28567 external_mini_cluster.cc:1428] Setting key 8732bcb72321677c9e9f178ff1d2cefa
Sep 28 02:52:32 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563952, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:32.902410 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:32.883167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:46781 (local address 127.27.229.254:46367)
0928 02:52:32.883359 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:32.883363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:32.884543 (+ 1180us) server_negotiation.cc:407] Connection header received
0928 02:52:32.886398 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:32.886403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:32.886497 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:32.886607 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:32.888406 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.889437 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.891562 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.892339 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.892483 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:32.896170 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:32.896175 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:32.896178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:32.896209 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:32.898995 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:32.900034 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:32.900038 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:32.900040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:32.900164 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:32.900679 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:32.900682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:32.900684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:32.900938 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:32.901047 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:32.902000 (+ 953us) server_negotiation.cc:299] Negotiation successful
0928 02:52:32.902170 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:52:32.904139 32504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:32.883631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46367 (local address 127.27.229.194:46781)
0928 02:52:32.884366 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:32.884414 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:32.886109 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:32.886909 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:32.886917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:32.887435 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:32.888180 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:32.888206 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.889615 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.889623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:32.891410 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:32.891423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:32.892569 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:32.892574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:32.892770 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:32.893538 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:32.893567 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:32.895905 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:32.899142 (+ 3237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:32.899156 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:32.899179 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:32.899884 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:32.900284 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:32.900300 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:32.900308 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:32.900566 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:32.901069 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:32.901080 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:32.901574 (+ 494us) client_negotiation.cc:769] Sending connection context
0928 02:52:32.901910 (+ 336us) client_negotiation.cc:240] Negotiation successful
0928 02:52:32.902547 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":325,"threads_started":1}
I20260928 02:52:32.905638 32502 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46367
I20260928 02:52:32.905956 32502 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:32.906761 32502 heartbeater.cc:511] Master 127.27.229.254:46367 requested a full tablet report, sending...
I20260928 02:52:32.908193 32179 ts_manager.cc:194] Registered new tserver with Master: 9553126427cb45fd9ce2df66fdeb425b (127.27.229.194:40529)
I20260928 02:52:32.908988 32179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:46781
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:32.938443 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:42877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:33.265978 32507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:33.266265 32507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:33.266423 32507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:33.273686 32507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:33.273804 32507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:33.273988 32507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:33.283968 32507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46367
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:33.285310 32507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:33.286643 32507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:33.300431 32515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:33.300498 32513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:33.300519 32512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:33.300805 32507 server_base.cc:1034] running on GCE node
I20260928 02:52:33.301688 32507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:33.302726 32507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:33.303979 32507 hybrid_clock.cc:648] HybridClock initialized: now 1790563953303922 us; error 46 us; skew 500 ppm
I20260928 02:52:33.304342 32507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:33.307729 32507 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:33.309202 32507 webserver.cc:460] Webserver started at http://127.27.229.195:45323/ using document root <none> and password file <none>
I20260928 02:52:33.309937 32507 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:33.310074 32507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:33.310386 32507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:33.313393 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "5921e2efefc44304b1f69e1b71ef50da"
format_stamp: "Formatted at 2026-09-28 02:52:33 on dist-test-slave-s1kg"
server_key: "0a655a35349a4dcb86845e2c443792eb"
server_key_iv: "796954ae2a16bc41978d59e5186fe95c"
server_key_version: "encryptionkey@0"
I20260928 02:52:33.314182 32507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "5921e2efefc44304b1f69e1b71ef50da"
format_stamp: "Formatted at 2026-09-28 02:52:33 on dist-test-slave-s1kg"
server_key: "0a655a35349a4dcb86845e2c443792eb"
server_key_iv: "796954ae2a16bc41978d59e5186fe95c"
server_key_version: "encryptionkey@0"
I20260928 02:52:33.319474 32507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:33.323057 32522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:33.324419 32507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:33.324601 32507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "5921e2efefc44304b1f69e1b71ef50da"
format_stamp: "Formatted at 2026-09-28 02:52:33 on dist-test-slave-s1kg"
server_key: "0a655a35349a4dcb86845e2c443792eb"
server_key_iv: "796954ae2a16bc41978d59e5186fe95c"
server_key_version: "encryptionkey@0"
I20260928 02:52:33.324820 32507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:33.342590 32507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:33.347251 32507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:33.347554 32507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:33.348816 32507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:33.351518 32507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:33.351622 32507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:33.351751 32507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:33.351840 32507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:33.390004 32507 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45963
I20260928 02:52:33.390048 32634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45963 every 8 connection(s)
I20260928 02:52:33.391453 32507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:33.398973 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32507
I20260928 02:52:33.399168 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:33.399577 28567 external_mini_cluster.cc:1428] Setting key 204f701f1eb067e1acae74066e1db8c1
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:33.411332 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.393910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:45925 (local address 127.27.229.254:46367)
0928 02:52:33.394091 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.394095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.395260 (+ 1165us) server_negotiation.cc:407] Connection header received
0928 02:52:33.397048 (+ 1788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.397053 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.397170 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.397285 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.398701 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.399667 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.401625 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.402037 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.402164 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.405725 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.405730 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.405733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.405764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.408347 (+ 2583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.409060 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.409064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.409066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.409182 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.409652 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.409656 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.409659 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.409942 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.410061 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.410883 (+ 822us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.411062 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:52:33.412267 32637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.394295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46367 (local address 127.27.229.195:45925)
0928 02:52:33.395066 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:33.395132 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:33.396791 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:33.397521 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:33.397528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:33.398008 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:33.398539 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:33.398555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.399813 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.399820 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:33.401495 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:33.401513 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.402262 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.402272 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:33.402552 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.403272 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:33.403298 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:33.405533 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:33.408502 (+ 2969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:33.408509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:33.408522 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:33.408955 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:33.409289 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:33.409292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:33.409295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:33.409508 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:33.410064 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:33.410070 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:33.410398 (+ 328us) client_negotiation.cc:769] Sending connection context
0928 02:52:33.410637 (+ 239us) client_negotiation.cc:240] Negotiation successful
0928 02:52:33.411144 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":316,"threads_started":1}
I20260928 02:52:33.413965 32635 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46367
I20260928 02:52:33.414340 32635 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:33.415225 32635 heartbeater.cc:511] Master 127.27.229.254:46367 requested a full tablet report, sending...
I20260928 02:52:33.416433 32368 heartbeater.cc:503] Master 127.27.229.254:46367 was elected leader, sending a full tablet report...
I20260928 02:52:33.416646 32179 ts_manager.cc:194] Registered new tserver with Master: 5921e2efefc44304b1f69e1b71ef50da (127.27.229.195:45963)
I20260928 02:52:33.417800 32179 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:45925
I20260928 02:52:33.427668 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:33.477911 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.461986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.27.229.193:39001)
0928 02:52:33.462652 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.462659 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.462687 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:52:33.462760 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.462766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.463058 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.463196 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.464697 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.465825 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.467215 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.467799 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.467920 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.472060 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.472074 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.472088 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.472133 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.475281 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.475921 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.475929 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.475931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.476055 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.476395 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.476403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.476406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.476840 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.477075 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.477349 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.477569 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":361,"threads_started":1}
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:33.496861 32370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.482527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35862 (local address 127.27.229.254:46367)
0928 02:52:33.482742 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.482748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.482765 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:52:33.482967 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.482971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.483031 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.483114 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.484380 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.485136 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.486409 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.486990 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.487093 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.491309 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.491314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.491317 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.491353 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.493950 (+ 2597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.494611 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.494615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.494618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.494741 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.495259 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.495263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.495265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.495696 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.495887 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.496305 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.496485 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 02:52:33.497520 32179 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35862
I20260928 02:52:33.498631 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32240
I20260928 02:52:33.512107 32642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.499580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42388 (local address 127.27.229.193:39001)
0928 02:52:33.499855 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.499860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.500080 (+ 220us) server_negotiation.cc:407] Connection header received
0928 02:52:33.500288 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.500292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.500362 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.500470 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.501701 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.502392 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.503688 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.504115 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.504250 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.506641 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.506647 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.506650 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.506682 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.509419 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.510134 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.510141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.510144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.510278 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.510661 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.510667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.510670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.510997 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.511248 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.511547 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.511775 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 02:52:33.512843 32363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:33.596024 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32240
I20260928 02:52:33.613595 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32374
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:33.629664 32647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.614620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52396 (local address 127.27.229.194:40529)
0928 02:52:33.615176 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.615182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.615206 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:52:33.615284 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.615290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.615512 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.615690 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.616828 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.617818 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.619188 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.619716 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.619853 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.624046 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.624057 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.624071 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.624110 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.626963 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.627742 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.627754 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.627757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.627890 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.628281 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.628285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.628287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.628625 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.628952 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.629127 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.629384 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":321,"threads_started":1}
I20260928 02:52:33.631228 32497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:33.712810 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32374
I20260928 02:52:33.728655 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32507
Sep 28 02:52:33 dist-test-slave-s1kg krb5kdc[32134](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563953, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:33.745251 32649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:33.729682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43860 (local address 127.27.229.195:45963)
0928 02:52:33.730282 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:33.730289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:33.730317 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:52:33.730393 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:33.730400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:33.730602 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:33.730728 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:33.732070 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.733144 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.734326 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:33.734943 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:33.735038 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:33.739099 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:33.739111 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:33.739125 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:33.739161 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:33.742337 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.743226 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.743234 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.743238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.743368 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:33.743728 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:33.743732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:33.743734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:33.744123 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:33.744348 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:33.744713 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:52:33.744943 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":327,"threads_started":1}
I20260928 02:52:33.746555 32630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:33.836035 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32507
I20260928 02:52:33.854450 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32147
I20260928 02:52:33.855767 32209 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:33.957963 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32147
2026-09-28T02:52:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3891 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:34 dist-test-slave-s1kg krb5kdc[32654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:34 dist-test-slave-s1kg krb5kdc[32654](info): set up 2 sockets
Sep 28 02:52:34 dist-test-slave-s1kg krb5kdc[32654](info): commencing operation
krb5kdc: starting...
W20260928 02:52:35.047410 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:35 dist-test-slave-s1kg krb5kdc[32654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:35.258414 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39777
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:35.592482 32667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:35.592856 32667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:35.593026 32667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:35.600519 32667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:35.600663 32667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:35.600740 32667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:35.600847 32667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:35.600922 32667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:35.611086 32667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:35.612409 32667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:35.613704 32667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:35.625789 32672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:35.625782 32673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:35.625972 32675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:35.626600 32667 server_base.cc:1034] running on GCE node
I20260928 02:52:35.627507 32667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:35.629083 32667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:35.630338 32667 hybrid_clock.cc:648] HybridClock initialized: now 1790563955630298 us; error 64 us; skew 500 ppm
I20260928 02:52:35.630710 32667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:35 dist-test-slave-s1kg krb5kdc[32654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:35.633944 32667 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:35.635226 32667 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:35.635370 32667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:35.635704 32667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:35.638708 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "6df5ba46e65344b0b0452b2e0ce1018f"
format_stamp: "Formatted at 2026-09-28 02:52:35 on dist-test-slave-s1kg"
server_key: "8dbc2c02013c5b9d224a358a25e165f4"
server_key_iv: "d534951a3212d541c0158019c19d4059"
server_key_version: "encryptionkey@0"
I20260928 02:52:35.639586 32667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "6df5ba46e65344b0b0452b2e0ce1018f"
format_stamp: "Formatted at 2026-09-28 02:52:35 on dist-test-slave-s1kg"
server_key: "8dbc2c02013c5b9d224a358a25e165f4"
server_key_iv: "d534951a3212d541c0158019c19d4059"
server_key_version: "encryptionkey@0"
I20260928 02:52:35.644820 32667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:35.648707 32681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:35.650096 32667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:35.650301 32667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "6df5ba46e65344b0b0452b2e0ce1018f"
format_stamp: "Formatted at 2026-09-28 02:52:35 on dist-test-slave-s1kg"
server_key: "8dbc2c02013c5b9d224a358a25e165f4"
server_key_iv: "d534951a3212d541c0158019c19d4059"
server_key_version: "encryptionkey@0"
I20260928 02:52:35.650537 32667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:35.681609 32667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:35.686235 32667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:35.686540 32667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:35.707088 32667 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:39777
I20260928 02:52:35.707075 32732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:39777 every 8 connection(s)
I20260928 02:52:35.708110 32667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:35.709056 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32667
I20260928 02:52:35.709337 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:35.710230 28567 external_mini_cluster.cc:1428] Setting key a79606282b1671b708601fa00fcb4fde
I20260928 02:52:35.713394 32733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:35.724603 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: Bootstrap starting.
Sep 28 02:52:35 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:35.728389 32733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:35.729507 32733 log.cc:824] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:35.732600 32733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: No bootstrap required, opened a new log
I20260928 02:52:35.733222 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:35.712431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.27.229.254:39777)
0928 02:52:35.713304 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:35.713318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:35.713361 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:52:35.715166 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:35.715191 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:35.715601 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:35.716035 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:35.717652 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:35.719085 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:35.720514 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:35.721265 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:35.721499 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:35.725701 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:35.725710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:35.725726 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:35.725771 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:35.728969 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:35.729723 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:35.729730 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:35.729736 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:35.729916 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:35.730312 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:35.730316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:35.730320 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:35.730988 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:35.731275 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:35.731767 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:52:35.732193 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":666,"thread_start_us":444,"threads_started":1}
I20260928 02:52:35.741276 32733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } }
I20260928 02:52:35.741677 32733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:35.741789 32733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6df5ba46e65344b0b0452b2e0ce1018f, State: Initialized, Role: FOLLOWER
I20260928 02:52:35.742339 32733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } }
I20260928 02:52:35.742609 32733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:35.742765 32733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:35.742990 32733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:35.745160 32733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } }
I20260928 02:52:35.745682 32733 leader_election.cc:302] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6df5ba46e65344b0b0452b2e0ce1018f; no voters:
I20260928 02:52:35.746315 32733 leader_election.cc:288] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:35.746492 32738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:35.747946 32738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [term 1 LEADER]: Becoming Leader. State: Replica: 6df5ba46e65344b0b0452b2e0ce1018f, State: Running, Role: LEADER
I20260928 02:52:35.748482 32738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } }
I20260928 02:52:35.748678 32733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:35.751466 32739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } } }
I20260928 02:52:35.751825 32739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:35.751516 32740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6df5ba46e65344b0b0452b2e0ce1018f. Latest consensus state: current_term: 1 leader_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6df5ba46e65344b0b0452b2e0ce1018f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39777 } } }
I20260928 02:52:35.751971 32740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:35.754530 32747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:35.759749 32747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:35.769213 32747 catalog_manager.cc:1347] Generated new cluster ID: c0da5ff9f2e24db19a5133658e74d3f7
I20260928 02:52:35.769349 32747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:35.790076 32747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:35.791746 32747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:35.799618 32747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: Generated new TSK 0
I20260928 02:52:35.800714 32747 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:35.851471 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:36.186242 32759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:36.186583 32759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:36.186757 32759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:36.194058 32759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:36.194221 32759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:36.194411 32759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:36.204439 32759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:36.205763 32759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:36.207062 32759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:36.219769 32764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:36.220057 32765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:36.220180 32767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:36.221269 32759 server_base.cc:1034] running on GCE node
I20260928 02:52:36.221841 32759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:36.222803 32759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:36.224066 32759 hybrid_clock.cc:648] HybridClock initialized: now 1790563956224020 us; error 41 us; skew 500 ppm
I20260928 02:52:36.224397 32759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:36 dist-test-slave-s1kg krb5kdc[32654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:36.227703 32759 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:36.228935 32759 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:36.229070 32759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:36.229393 32759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:36.232429 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "be306ebe19964a25b61159c738255e2d"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "763b36023e80e628b78a873f5f20ac01"
server_key_iv: "cdb2e1d2e4782c714afd83e7cc76d5ac"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.233296 32759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "be306ebe19964a25b61159c738255e2d"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "763b36023e80e628b78a873f5f20ac01"
server_key_iv: "cdb2e1d2e4782c714afd83e7cc76d5ac"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.238276 32759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:52:36.241876 305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.243438 32759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:36.243624 32759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "be306ebe19964a25b61159c738255e2d"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "763b36023e80e628b78a873f5f20ac01"
server_key_iv: "cdb2e1d2e4782c714afd83e7cc76d5ac"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.243865 32759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:36.259994 32759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:36.264613 32759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:36.264905 32759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:36.265859 32759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:36.267328 32759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:36.267453 32759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.267578 32759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:36.267669 32759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.302023 32759 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37179
I20260928 02:52:36.302067 417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37179 every 8 connection(s)
I20260928 02:52:36.303006 32759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:36.304224 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 32759
I20260928 02:52:36.304456 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:36.305181 28567 external_mini_cluster.cc:1428] Setting key 5c111c2814aacc029da0ad15750a862b
Sep 28 02:52:36 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:36.326736 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:36.305593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:58853 (local address 127.27.229.254:39777)
0928 02:52:36.312288 (+ 6695us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:36.312296 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:36.312321 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:52:36.312387 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:36.312394 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:36.312480 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:36.312604 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:36.314203 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.315005 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.316574 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.316961 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.317099 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:36.320685 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:36.320690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:36.320693 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:36.320724 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:36.323515 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:36.324230 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:36.324236 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:36.324240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:36.324417 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:36.324893 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:36.324900 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:36.324902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:36.325270 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:36.325424 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:36.326137 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 02:52:36.326353 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6553,"thread_start_us":6444,"threads_started":1}
I20260928 02:52:36.327765 421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:36.306239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39777 (local address 127.27.229.193:58853)
0928 02:52:36.307290 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:36.307357 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:36.310615 (+ 3258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:36.312836 (+ 2221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:36.312846 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:36.313419 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:36.314014 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:36.314030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.315158 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.315163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:36.316415 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:36.316427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.317212 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.317218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:36.317444 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:36.318153 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:36.318181 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:36.320443 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:36.323658 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:36.323667 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:36.323685 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:36.324107 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:36.324583 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:36.324588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:36.324592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:36.324779 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:36.325401 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:36.325409 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:36.325757 (+ 348us) client_negotiation.cc:769] Sending connection context
0928 02:52:36.326037 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:52:36.326564 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":386,"threads_started":1}
I20260928 02:52:36.329442 418 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39777
I20260928 02:52:36.329756 418 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:36.330444 418 heartbeater.cc:511] Master 127.27.229.254:39777 requested a full tablet report, sending...
I20260928 02:52:36.332513 32698 ts_manager.cc:194] Registered new tserver with Master: be306ebe19964a25b61159c738255e2d (127.27.229.193:37179)
I20260928 02:52:36.333966 32698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:58853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:36.351404 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:36.677664 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:36.677964 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:36.678064 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:36.685774 424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:36.685916 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:36.686066 424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:36.696153 424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:36.697448 424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:36.698740 424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:36.715582 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:36.715730 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:36.716416 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:36.717165 424 server_base.cc:1034] running on GCE node
I20260928 02:52:36.717859 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:36.718967 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:36.720219 424 hybrid_clock.cc:648] HybridClock initialized: now 1790563956720159 us; error 54 us; skew 500 ppm
I20260928 02:52:36.720561 424 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:36 dist-test-slave-s1kg krb5kdc[32654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:36.724000 424 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:36.725270 424 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:36.725361 424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:36.725713 424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:36.728802 424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "111fc3a9ce9c48439277df2adde46975"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "fbb71bd5b4088ce79ea59cd70e65700d"
server_key_iv: "e02997fc6d25974e8e387f46667690a6"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.729658 424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "111fc3a9ce9c48439277df2adde46975"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "fbb71bd5b4088ce79ea59cd70e65700d"
server_key_iv: "e02997fc6d25974e8e387f46667690a6"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.735277 424 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:52:36.738952 438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.740406 424 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:36.740607 424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "111fc3a9ce9c48439277df2adde46975"
format_stamp: "Formatted at 2026-09-28 02:52:36 on dist-test-slave-s1kg"
server_key: "fbb71bd5b4088ce79ea59cd70e65700d"
server_key_iv: "e02997fc6d25974e8e387f46667690a6"
server_key_version: "encryptionkey@0"
I20260928 02:52:36.740821 424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:36.763815 424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:36.768520 424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:36.768849 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:36.769910 424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:36.771534 424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:36.771627 424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.771953 424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:36.772100 424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:36.810917 424 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:35227
I20260928 02:52:36.811014 550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:35227 every 8 connection(s)
I20260928 02:52:36.811802 424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:36.813241 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 424
I20260928 02:52:36.813460 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:36.813916 28567 external_mini_cluster.cc:1428] Setting key d19d31ff9e22a6cdb48fb6fd244f5a27
Sep 28 02:52:36 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563956, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:52:36.833038 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:36.814958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33931 (local address 127.27.229.254:39777)
0928 02:52:36.815156 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:36.815162 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:36.816501 (+ 1339us) server_negotiation.cc:407] Connection header received
0928 02:52:36.818374 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:36.818380 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:36.818473 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:36.818592 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:36.820235 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.820937 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.822545 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.823070 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.823207 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:36.826979 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:36.826984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:36.826988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:36.827021 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:36.829764 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:36.830572 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:36.830576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:36.830578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:36.830701 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:36.831180 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:36.831191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:36.831194 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:36.831461 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:36.831616 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:36.832416 (+ 800us) server_negotiation.cc:299] Negotiation successful
0928 02:52:36.832649 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:36.834098 553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:36.815574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39777 (local address 127.27.229.194:33931)
0928 02:52:36.816348 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:36.816393 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:36.818084 (+ 1691us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:36.818885 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:36.818897 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:36.819457 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:36.820047 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:36.820068 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.821100 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.821110 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:36.822391 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:36.822402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:36.823315 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:36.823322 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:36.823544 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:36.824260 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:36.824292 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:36.826682 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:36.829934 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:36.829945 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:36.829965 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:36.830426 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:36.830820 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:36.830861 (+ 41us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:36.830865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:36.831054 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:36.831593 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:36.831602 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:36.831971 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:52:36.832299 (+ 328us) client_negotiation.cc:240] Negotiation successful
0928 02:52:36.832882 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":344,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:52:36.835973 551 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39777
I20260928 02:52:36.836329 551 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:36.837426 551 heartbeater.cc:511] Master 127.27.229.254:39777 requested a full tablet report, sending...
I20260928 02:52:36.839162 32698 ts_manager.cc:194] Registered new tserver with Master: 111fc3a9ce9c48439277df2adde46975 (127.27.229.194:35227)
I20260928 02:52:36.839993 32698 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:33931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:36.862383 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:37.199542 556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:37.199899 556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:37.200007 556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:37.207471 556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:37.207600 556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:37.207721 556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:37.218919 556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:37.220263 556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:37.221539 556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:37.236073 562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:37.236158 561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:37.236135 564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:37.236965 556 server_base.cc:1034] running on GCE node
I20260928 02:52:37.237727 556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:37.239051 556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:37.240509 556 hybrid_clock.cc:648] HybridClock initialized: now 1790563957240383 us; error 138 us; skew 500 ppm
I20260928 02:52:37.240880 556 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:37 dist-test-slave-s1kg krb5kdc[32654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:37.244809 556 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:37.246579 556 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:37.246728 556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:37.247206 556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:37.250399 556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "7ac5a0bd18d04eaba8425e8ccca88c29"
format_stamp: "Formatted at 2026-09-28 02:52:37 on dist-test-slave-s1kg"
server_key: "316249ab339585a3f2053757348c8394"
server_key_iv: "32856a3927fbe7efc96a37d0c648849d"
server_key_version: "encryptionkey@0"
I20260928 02:52:37.251259 556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "7ac5a0bd18d04eaba8425e8ccca88c29"
format_stamp: "Formatted at 2026-09-28 02:52:37 on dist-test-slave-s1kg"
server_key: "316249ab339585a3f2053757348c8394"
server_key_iv: "32856a3927fbe7efc96a37d0c648849d"
server_key_version: "encryptionkey@0"
I20260928 02:52:37.256832 556 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:37.260725 570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:37.262596 556 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:37.262813 556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "7ac5a0bd18d04eaba8425e8ccca88c29"
format_stamp: "Formatted at 2026-09-28 02:52:37 on dist-test-slave-s1kg"
server_key: "316249ab339585a3f2053757348c8394"
server_key_iv: "32856a3927fbe7efc96a37d0c648849d"
server_key_version: "encryptionkey@0"
I20260928 02:52:37.263224 556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:37.275611 556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:37.280148 556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:37.280454 556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:37.281486 556 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:37.282930 556 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:37.283049 556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:37.283178 556 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:37.283277 556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:37.318545 556 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:46347
I20260928 02:52:37.318583 682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:46347 every 8 connection(s)
I20260928 02:52:37.319528 556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:37.323261 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 556
I20260928 02:52:37.323505 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:37.324003 28567 external_mini_cluster.cc:1428] Setting key 1b48638119bfaf89d82f1d7d1ea6a9be
Sep 28 02:52:37 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563957, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:37.337123 418 heartbeater.cc:503] Master 127.27.229.254:39777 was elected leader, sending a full tablet report...
I20260928 02:52:37.342197 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.322114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:42145 (local address 127.27.229.254:39777)
0928 02:52:37.322306 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.322311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.323788 (+ 1477us) server_negotiation.cc:407] Connection header received
0928 02:52:37.326030 (+ 2242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.326036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.326131 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.326227 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:37.328089 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.328929 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.330544 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.331148 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.331285 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.335496 (+ 4211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:37.335501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:37.335505 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:37.335543 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:37.339163 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.339961 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.339973 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.339976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.340102 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.340583 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.340587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.340589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.340874 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:37.341008 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.341779 (+ 771us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.341954 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:52:37.343493 685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.322632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39777 (local address 127.27.229.195:42145)
0928 02:52:37.323597 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.323645 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.325741 (+ 2096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.326520 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.326529 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.327160 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:37.327910 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.327930 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.329089 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.329093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.330372 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.330384 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.331652 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.331673 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.331989 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.332951 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:37.332984 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:37.335274 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:37.339324 (+ 4050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:37.339332 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:37.339348 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:37.339816 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:37.340224 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:37.340228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:37.340230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:37.340432 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:37.341003 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:37.341010 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:37.341409 (+ 399us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.341657 (+ 248us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.342094 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"thread_start_us":461,"threads_started":1}
I20260928 02:52:37.345767 683 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39777
I20260928 02:52:37.346139 683 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:37.346822 683 heartbeater.cc:511] Master 127.27.229.254:39777 requested a full tablet report, sending...
I20260928 02:52:37.348605 32697 ts_manager.cc:194] Registered new tserver with Master: 7ac5a0bd18d04eaba8425e8ccca88c29 (127.27.229.195:46347)
I20260928 02:52:37.349565 32697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:42145
I20260928 02:52:37.352682 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:52:37.374979 420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.357491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33542 (local address 127.27.229.254:39777)
0928 02:52:37.357672 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.357676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.358305 (+ 629us) server_negotiation.cc:407] Connection header received
0928 02:52:37.358466 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.358470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.358534 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.358627 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:37.361698 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.362954 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.365032 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.365816 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.365982 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.369457 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:37.369464 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:37.369470 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:37.369521 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:37.372180 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.372899 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.372905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.372909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.373048 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.373459 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.373464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.373468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.373808 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:37.374082 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.374406 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.374625 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:52:37.379631 32697 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:37.386700 32697 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:37.413177 696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.401573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:46347 (local address 127.0.0.1:48694)
0928 02:52:37.404267 (+ 2694us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.404285 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.404543 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.405481 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.405486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.405509 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.405940 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.405953 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.407985 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.407993 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.410085 (+ 2092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.410099 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.412114 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.412124 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.412348 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.412405 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.412566 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.412767 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"mutex_wait_us":27,"thread_start_us":2386,"threads_started":1}
I20260928 02:52:37.413211 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.402223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48694 (local address 127.27.229.195:46347)
0928 02:52:37.403791 (+ 1568us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.403802 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.404406 (+ 604us) server_negotiation.cc:407] Connection header received
0928 02:52:37.404750 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.404758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.404972 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.405136 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.406178 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.407801 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.410275 (+ 2474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.411878 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.412107 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.412294 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.412652 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.412838 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":895,"threads_started":1}
I20260928 02:52:37.414687 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.401739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42998 (local address 127.27.229.193:37179)
0928 02:52:37.404066 (+ 2327us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.404074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.404108 (+ 34us) server_negotiation.cc:407] Connection header received
0928 02:52:37.404220 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.404230 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.404554 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.405494 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.405751 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.407457 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.411476 (+ 4019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.413217 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.413442 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.413643 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.414110 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.414298 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2064,"thread_start_us":1788,"threads_started":1}
I20260928 02:52:37.415048 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.401418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37179 (local address 127.0.0.1:42998)
0928 02:52:37.402495 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.402547 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.402731 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.404784 (+ 2053us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.404792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.404821 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.405314 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.405328 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.407611 (+ 2283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.407617 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.411254 (+ 3637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.411269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.413700 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.413709 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.413911 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.413957 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.414060 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.414521 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":516,"thread_start_us":1065,"threads_started":1}
I20260928 02:52:37.417987 618 tablet_service.cc:1467] Processing CreateTablet for tablet 59973aa347134c3a8604476759ca8751 (DEFAULT_TABLE table=test-table [id=c46a33f2650a46e3a36b13b3db07f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:37.418566 353 tablet_service.cc:1467] Processing CreateTablet for tablet 59973aa347134c3a8604476759ca8751 (DEFAULT_TABLE table=test-table [id=c46a33f2650a46e3a36b13b3db07f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:37.419515 618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59973aa347134c3a8604476759ca8751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:37.420219 353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59973aa347134c3a8604476759ca8751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:37.421093 699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.404854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35227 (local address 127.0.0.1:37188)
0928 02:52:37.410699 (+ 5845us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.410723 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.410883 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.411782 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.411788 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.411817 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.412274 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.412287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.414157 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.414162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.417448 (+ 3286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.417467 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.420212 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.420218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.420393 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.420425 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.420539 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.420684 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5699,"mutex_wait_us":1256,"thread_start_us":5544,"threads_started":1}
I20260928 02:52:37.421154 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.405411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.27.229.194:35227)
0928 02:52:37.408956 (+ 3545us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.408966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.411014 (+ 2048us) server_negotiation.cc:407] Connection header received
0928 02:52:37.411150 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.411158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.411434 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.411660 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.412446 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.413952 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.417677 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.419694 (+ 2017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.419900 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.420085 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.420621 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.420802 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3361,"thread_start_us":3239,"threads_started":1}
I20260928 02:52:37.425625 486 tablet_service.cc:1467] Processing CreateTablet for tablet 59973aa347134c3a8604476759ca8751 (DEFAULT_TABLE table=test-table [id=c46a33f2650a46e3a36b13b3db07f19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:37.427296 486 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59973aa347134c3a8604476759ca8751. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:37.432806 701 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Bootstrap starting.
I20260928 02:52:37.436025 701 tablet_bootstrap.cc:654] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:37.436635 702 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Bootstrap starting.
I20260928 02:52:37.437876 701 log.cc:824] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:37.440237 702 tablet_bootstrap.cc:654] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:37.441566 701 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: No bootstrap required, opened a new log
I20260928 02:52:37.441572 702 log.cc:824] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:37.441969 701 ts_tablet_manager.cc:1397] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260928 02:52:37.443895 705 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Bootstrap starting.
I20260928 02:52:37.445056 702 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: No bootstrap required, opened a new log
I20260928 02:52:37.445334 702 ts_tablet_manager.cc:1397] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:52:37.448057 705 tablet_bootstrap.cc:654] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:37.449483 705 log.cc:824] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:37.452488 705 tablet_bootstrap.cc:492] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: No bootstrap required, opened a new log
I20260928 02:52:37.452781 705 ts_tablet_manager.cc:1397] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 02:52:37.453011 701 raft_consensus.cc:348] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.453440 701 raft_consensus.cc:374] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:37.453605 701 raft_consensus.cc:729] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac5a0bd18d04eaba8425e8ccca88c29, State: Initialized, Role: FOLLOWER
I20260928 02:52:37.454264 701 consensus_queue.cc:260] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.456625 702 raft_consensus.cc:348] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.457185 702 raft_consensus.cc:374] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:37.457293 702 raft_consensus.cc:729] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be306ebe19964a25b61159c738255e2d, State: Initialized, Role: FOLLOWER
I20260928 02:52:37.458160 702 consensus_queue.cc:260] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.460578 702 ts_tablet_manager.cc:1428] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260928 02:52:37.461390 683 heartbeater.cc:503] Master 127.27.229.254:39777 was elected leader, sending a full tablet report...
I20260928 02:52:37.462325 701 ts_tablet_manager.cc:1428] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Time spent starting tablet: real 0.020s user 0.013s sys 0.006s
I20260928 02:52:37.463557 705 raft_consensus.cc:348] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.464107 705 raft_consensus.cc:374] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:37.464227 705 raft_consensus.cc:729] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 111fc3a9ce9c48439277df2adde46975, State: Initialized, Role: FOLLOWER
I20260928 02:52:37.465009 705 consensus_queue.cc:260] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.466256 551 heartbeater.cc:503] Master 127.27.229.254:39777 was elected leader, sending a full tablet report...
I20260928 02:52:37.467113 705 ts_tablet_manager.cc:1428] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Time spent starting tablet: real 0.014s user 0.006s sys 0.009s
W20260928 02:52:37.555541 419 tablet.cc:2369] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:37.564055 552 tablet.cc:2369] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:37.571107 684 tablet.cc:2369] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:37.709360 708 raft_consensus.cc:482] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:37.709690 708 raft_consensus.cc:504] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.711761 708 leader_election.cc:288] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac5a0bd18d04eaba8425e8ccca88c29 (127.27.229.195:46347), 111fc3a9ce9c48439277df2adde46975 (127.27.229.194:35227)
I20260928 02:52:37.719707 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.712343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:52353 (local address 127.27.229.194:35227)
0928 02:52:37.712548 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.712554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.713341 (+ 787us) server_negotiation.cc:407] Connection header received
0928 02:52:37.713590 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.713595 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.713683 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.713809 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.714450 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.715577 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.717892 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.718685 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.718910 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.718988 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.719272 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.719397 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:52:37.719882 711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.712049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:35227 (local address 127.27.229.193:52353)
0928 02:52:37.713236 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.713253 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.713439 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.713825 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.713830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.713860 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.714282 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.714295 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.715723 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.715728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.717734 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.717744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.718927 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.718933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.719095 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.719121 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.719238 (+ 117us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.719385 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":271,"thread_start_us":447,"threads_started":1}
I20260928 02:52:37.720050 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.712204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:46217 (local address 127.27.229.195:46347)
0928 02:52:37.712391 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.712398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.713148 (+ 750us) server_negotiation.cc:407] Connection header received
0928 02:52:37.713392 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.713396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.713454 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.713554 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.714154 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.715057 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.717634 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.718920 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.719065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.719131 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.719669 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.719812 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:52:37.720777 710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.712049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:46347 (local address 127.27.229.193:46217)
0928 02:52:37.713020 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.713038 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.713227 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.713596 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.713600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.713623 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.713994 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.714006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.715194 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.715202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.717456 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.717471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.719280 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.719287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.719471 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.719516 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.720272 (+ 756us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.720412 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":51,"thread_start_us":488,"threads_started":1}
I20260928 02:52:37.720908 709 raft_consensus.cc:482] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:37.721306 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "be306ebe19964a25b61159c738255e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111fc3a9ce9c48439277df2adde46975" is_pre_election: true
I20260928 02:52:37.721218 709 raft_consensus.cc:504] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.721906 638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "be306ebe19964a25b61159c738255e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" is_pre_election: true
I20260928 02:52:37.722038 506 raft_consensus.cc:2454] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be306ebe19964a25b61159c738255e2d in term 0.
I20260928 02:52:37.722342 638 raft_consensus.cc:2454] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be306ebe19964a25b61159c738255e2d in term 0.
I20260928 02:52:37.723107 307 leader_election.cc:302] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ac5a0bd18d04eaba8425e8ccca88c29, be306ebe19964a25b61159c738255e2d; no voters:
I20260928 02:52:37.723340 709 leader_election.cc:288] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be306ebe19964a25b61159c738255e2d (127.27.229.193:37179), 7ac5a0bd18d04eaba8425e8ccca88c29 (127.27.229.195:46347)
I20260928 02:52:37.723639 708 raft_consensus.cc:2781] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:37.723810 708 raft_consensus.cc:482] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:37.723917 708 raft_consensus.cc:3037] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:37.728037 708 raft_consensus.cc:504] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.729209 638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "be306ebe19964a25b61159c738255e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29"
I20260928 02:52:37.729449 638 raft_consensus.cc:3037] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:37.729521 506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "be306ebe19964a25b61159c738255e2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "111fc3a9ce9c48439277df2adde46975"
I20260928 02:52:37.729600 708 leader_election.cc:288] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [CANDIDATE]: Term 1 election: Requested vote from peers 7ac5a0bd18d04eaba8425e8ccca88c29 (127.27.229.195:46347), 111fc3a9ce9c48439277df2adde46975 (127.27.229.194:35227)
I20260928 02:52:37.729756 506 raft_consensus.cc:3037] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:37.733013 638 raft_consensus.cc:2454] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be306ebe19964a25b61159c738255e2d in term 1.
I20260928 02:52:37.733023 506 raft_consensus.cc:2454] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be306ebe19964a25b61159c738255e2d in term 1.
I20260928 02:52:37.733530 309 leader_election.cc:302] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 111fc3a9ce9c48439277df2adde46975, 7ac5a0bd18d04eaba8425e8ccca88c29, be306ebe19964a25b61159c738255e2d; no voters:
I20260928 02:52:37.733942 708 raft_consensus.cc:2781] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:37.734114 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.724549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:60623 (local address 127.27.229.195:46347)
0928 02:52:37.725419 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.725426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.726520 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 02:52:37.726774 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.726781 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.726894 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.726983 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.727736 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.728574 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.731431 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.732752 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.732892 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.732956 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.733710 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.733852 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733}
I20260928 02:52:37.734181 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.725224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:45345 (local address 127.27.229.193:37179)
0928 02:52:37.725419 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.725425 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.726230 (+ 805us) server_negotiation.cc:407] Connection header received
0928 02:52:37.726531 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.726537 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.726608 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.726655 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:37.727527 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.728665 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.731555 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.732752 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.732885 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.732956 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.733710 (+ 754us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.733848 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 02:52:37.734668 713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.723695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37179 (local address 127.27.229.194:45345)
0928 02:52:37.726112 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.726136 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.726365 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.726888 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.726893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.726918 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.727323 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.727341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.728779 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.728783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.731392 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.731405 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.733200 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.733213 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.733470 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.733575 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.733974 (+ 399us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.734121 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2247,"mutex_wait_us":171,"thread_start_us":2368,"threads_started":1}
I20260928 02:52:37.734771 712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.724014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:46347 (local address 127.27.229.194:60623)
0928 02:52:37.726418 (+ 2404us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:37.726436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:37.726619 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:37.727135 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:37.727139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:37.727162 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:37.727587 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.727595 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.728705 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.728713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.731217 (+ 2504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:37.731237 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.733224 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.733229 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:37.733471 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.733552 (+ 81us) client_negotiation.cc:769] Sending connection context
0928 02:52:37.734266 (+ 714us) client_negotiation.cc:240] Negotiation successful
0928 02:52:37.734402 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2271,"mutex_wait_us":97,"thread_start_us":1692,"threads_started":1}
I20260928 02:52:37.734948 708 raft_consensus.cc:686] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 1 LEADER]: Becoming Leader. State: Replica: be306ebe19964a25b61159c738255e2d, State: Running, Role: LEADER
I20260928 02:52:37.735822 708 consensus_queue.cc:237] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } }
I20260928 02:52:37.735901 373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "111fc3a9ce9c48439277df2adde46975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be306ebe19964a25b61159c738255e2d" is_pre_election: true
I20260928 02:52:37.736251 638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59973aa347134c3a8604476759ca8751" candidate_uuid: "111fc3a9ce9c48439277df2adde46975" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" is_pre_election: true
I20260928 02:52:37.736554 638 raft_consensus.cc:2379] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 111fc3a9ce9c48439277df2adde46975 in current term 1: Already voted for candidate be306ebe19964a25b61159c738255e2d in this term.
I20260928 02:52:37.737471 440 leader_election.cc:302] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 111fc3a9ce9c48439277df2adde46975; no voters: 7ac5a0bd18d04eaba8425e8ccca88c29, be306ebe19964a25b61159c738255e2d
I20260928 02:52:37.737849 709 raft_consensus.cc:2726] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:52:37.741149 32698 catalog_manager.cc:5495] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d reported cstate change: term changed from 0 to 1, leader changed from <none> to be306ebe19964a25b61159c738255e2d (127.27.229.193). New cstate: current_term: 1 leader_uuid: "be306ebe19964a25b61159c738255e2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be306ebe19964a25b61159c738255e2d" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37179 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:37.787322 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.781393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.27.229.193:37179)
0928 02:52:37.781653 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.781657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.781677 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:52:37.781924 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.781928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.782018 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.782150 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:37.782556 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.783444 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.784846 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.785308 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.785414 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.785993 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:37.786562 (+ 569us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:37.786675 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.786887 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.786961 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 02:52:37.795672 638 raft_consensus.cc:1261] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 1 FOLLOWER]: Refusing update from remote peer be306ebe19964a25b61159c738255e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:37.795711 506 raft_consensus.cc:1261] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 1 FOLLOWER]: Refusing update from remote peer be306ebe19964a25b61159c738255e2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:37.796725 708 consensus_queue.cc:1035] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac5a0bd18d04eaba8425e8ccca88c29" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 46347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:37.797150 714 consensus_queue.cc:1035] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "111fc3a9ce9c48439277df2adde46975" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 35227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:37.808748 717 mvcc.cc:205] Tried to move back new op lower bound from 7334149971120947200 to 7334149970893631488. Current Snapshot: MvccSnapshot[applied={T|T < 7334149971120947200}]
I20260928 02:52:37.809522 719 mvcc.cc:205] Tried to move back new op lower bound from 7334149971120947200 to 7334149970893631488. Current Snapshot: MvccSnapshot[applied={T|T < 7334149971120947200}]
I20260928 02:52:37.833966 32697 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:37.834244 32697 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:37.838327 32697 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6df5ba46e65344b0b0452b2e0ce1018f: Sending DeleteTablet for 3 replicas of tablet 59973aa347134c3a8604476759ca8751
I20260928 02:52:37.839542 353 tablet_service.cc:1514] Processing DeleteTablet for tablet 59973aa347134c3a8604476759ca8751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:42998
I20260928 02:52:37.839618 486 tablet_service.cc:1514] Processing DeleteTablet for tablet 59973aa347134c3a8604476759ca8751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:37188
I20260928 02:52:37.839618 618 tablet_service.cc:1514] Processing DeleteTablet for tablet 59973aa347134c3a8604476759ca8751 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:48694
I20260928 02:52:37.842592 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32759
I20260928 02:52:37.852650 729 tablet_replica.cc:330] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: stopping tablet replica
I20260928 02:52:37.854954 729 raft_consensus.cc:2229] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:37.856204 729 raft_consensus.cc:2258] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:37.860186 729 ts_tablet_manager.cc:1905] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:37.864508 727 tablet_replica.cc:330] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: stopping tablet replica
I20260928 02:52:37.865043 727 raft_consensus.cc:2229] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:37.865749 727 raft_consensus.cc:2258] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:37.867128 728 tablet_replica.cc:330] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: stopping tablet replica
I20260928 02:52:37.867771 728 raft_consensus.cc:2229] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:37.868387 728 raft_consensus.cc:2258] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:37.868938 727 ts_tablet_manager.cc:1905] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:37.871838 729 ts_tablet_manager.cc:1918] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:37.872102 729 log.cc:1196] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/59973aa347134c3a8604476759ca8751
I20260928 02:52:37.872388 728 ts_tablet_manager.cc:1905] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:37.872829 729 ts_tablet_manager.cc:1939] T 59973aa347134c3a8604476759ca8751 P 111fc3a9ce9c48439277df2adde46975: Deleting consensus metadata
I20260928 02:52:37.874953 32685 catalog_manager.cc:4841] TS 111fc3a9ce9c48439277df2adde46975 (127.27.229.194:35227): tablet 59973aa347134c3a8604476759ca8751 (table test-table [id=c46a33f2650a46e3a36b13b3db07f19f]) successfully deleted
I20260928 02:52:37.876552 727 ts_tablet_manager.cc:1918] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:37.876729 727 log.cc:1196] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/59973aa347134c3a8604476759ca8751
I20260928 02:52:37.877290 727 ts_tablet_manager.cc:1939] T 59973aa347134c3a8604476759ca8751 P be306ebe19964a25b61159c738255e2d: Deleting consensus metadata
I20260928 02:52:37.878624 728 ts_tablet_manager.cc:1918] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:37.878822 728 log.cc:1196] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/59973aa347134c3a8604476759ca8751
I20260928 02:52:37.879158 32682 catalog_manager.cc:4841] TS be306ebe19964a25b61159c738255e2d (127.27.229.193:37179): tablet 59973aa347134c3a8604476759ca8751 (table test-table [id=c46a33f2650a46e3a36b13b3db07f19f]) successfully deleted
I20260928 02:52:37.879472 728 ts_tablet_manager.cc:1939] T 59973aa347134c3a8604476759ca8751 P 7ac5a0bd18d04eaba8425e8ccca88c29: Deleting consensus metadata
I20260928 02:52:37.881496 32683 catalog_manager.cc:4841] TS 7ac5a0bd18d04eaba8425e8ccca88c29 (127.27.229.195:46347): tablet 59973aa347134c3a8604476759ca8751 (table test-table [id=c46a33f2650a46e3a36b13b3db07f19f]) successfully deleted
Sep 28 02:52:37 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:37.889745 697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:37.844237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43012 (local address 127.27.229.193:37179)
0928 02:52:37.844504 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:37.844512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:37.857969 (+ 13457us) server_negotiation.cc:407] Connection header received
0928 02:52:37.858497 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:37.858506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:37.858619 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:37.858707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:37.860758 (+ 2051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.862348 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.867125 (+ 4777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:37.867885 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:37.868016 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:37.883539 (+ 15523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:37.883552 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:37.883567 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:37.883609 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:37.886952 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.887721 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.887729 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.887732 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.887867 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:37.888250 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:37.888254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:37.888256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:37.888613 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:37.888963 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:37.889188 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:52:37.889412 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:52:37.890471 413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:37.991761 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32759
I20260928 02:52:38.010661 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 424
Sep 28 02:52:38 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:38.026532 700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:38.011570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37192 (local address 127.27.229.194:35227)
0928 02:52:38.011776 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:38.011780 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:38.011800 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:52:38.012026 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:38.012030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:38.012099 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:38.012199 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:38.013444 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:38.014206 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:38.015590 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:38.015994 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:38.016107 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:38.020493 (+ 4386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:38.020509 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:38.020526 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:38.020567 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:38.023560 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:38.024299 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:38.024307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:38.024309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:38.024462 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:38.024912 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:38.024917 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:38.024919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:38.025318 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:38.025558 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:38.025883 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:52:38.026139 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:52:38.027237 546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:38.136335 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 424
I20260928 02:52:38.153872 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 556
Sep 28 02:52:38 dist-test-slave-s1kg krb5kdc[32654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:38.169335 698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:38.154703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48708 (local address 127.27.229.195:46347)
0928 02:52:38.154902 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:38.154907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:38.155003 (+ 96us) server_negotiation.cc:407] Connection header received
0928 02:52:38.155201 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:38.155206 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:38.155284 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:38.155384 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:38.156640 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:38.157379 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:38.158662 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:38.159250 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:38.159350 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:38.163686 (+ 4336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:38.163701 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:38.163718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:38.163759 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:38.166418 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:38.167289 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:38.167296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:38.167299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:38.167444 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:38.167822 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:38.167826 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:38.167828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:38.168173 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:38.168502 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:38.168804 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 02:52:38.169026 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:52:38.169945 678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:38.259617 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 556
I20260928 02:52:38.276363 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 32667
I20260928 02:52:38.277622 32728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:38.387729 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 32667
2026-09-28T02:52:38Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4428 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T02:52:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:38Z Disabled control of system clock
I20260928 02:52:38.456722 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34133
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:52:38.790786 738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:38.791107 738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:38.791285 738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:38.798722 738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:38.798859 738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:38.798902 738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:38.798997 738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:38.799083 738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:38.808650 738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:38.809983 738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:38.811939 738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:38.823871 743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:38.823975 744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:38.824002 746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:38.825057 738 server_base.cc:1034] running on GCE node
I20260928 02:52:38.825887 738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:38.827387 738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:38.828603 738 hybrid_clock.cc:648] HybridClock initialized: now 1790563958828541 us; error 59 us; skew 500 ppm
I20260928 02:52:38.828930 738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:38.831462 738 webserver.cc:460] Webserver started at http://127.27.229.254:40001/ using document root <none> and password file <none>
I20260928 02:52:38.832150 738 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:38.832235 738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:38.832513 738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:38.835446 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "c201e316fcce4a8d8cebf3a466f6cc8f"
format_stamp: "Formatted at 2026-09-28 02:52:38 on dist-test-slave-s1kg"
server_key: "ea0da3593c1a7d3be0f09620ce51839a"
server_key_iv: "02b407e240c942ca1810fb1cea2d0493"
server_key_version: "encryptionkey@0"
I20260928 02:52:38.836253 738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "c201e316fcce4a8d8cebf3a466f6cc8f"
format_stamp: "Formatted at 2026-09-28 02:52:38 on dist-test-slave-s1kg"
server_key: "ea0da3593c1a7d3be0f09620ce51839a"
server_key_iv: "02b407e240c942ca1810fb1cea2d0493"
server_key_version: "encryptionkey@0"
I20260928 02:52:38.841424 738 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:38.844970 752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:38.846485 738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:38.846668 738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "c201e316fcce4a8d8cebf3a466f6cc8f"
format_stamp: "Formatted at 2026-09-28 02:52:38 on dist-test-slave-s1kg"
server_key: "ea0da3593c1a7d3be0f09620ce51839a"
server_key_iv: "02b407e240c942ca1810fb1cea2d0493"
server_key_version: "encryptionkey@0"
I20260928 02:52:38.846915 738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:38.855669 738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:38.856791 738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:38.857161 738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:38.876619 738 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34133
I20260928 02:52:38.876641 803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34133 every 8 connection(s)
I20260928 02:52:38.878075 738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:38.882314 804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:38.887117 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 738
I20260928 02:52:38.887307 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:38.887936 28567 external_mini_cluster.cc:1428] Setting key c027897316305711cadabc0ae47ba9b0
I20260928 02:52:38.894992 804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: Bootstrap starting.
I20260928 02:52:38.899035 807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:38.890375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50838 (local address 127.27.229.254:34133)
0928 02:52:38.891910 (+ 1535us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:38.891928 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:38.891997 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:52:38.894548 (+ 2551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:38.894583 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:38.895255 (+ 672us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:38.895748 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:38.896851 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:38.896872 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:38.896900 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:38.896923 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:38.897018 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:38.897046 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:38.897283 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:38.897428 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:52:38.897721 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":846,"thread_start_us":512,"threads_started":1}
I20260928 02:52:38.899437 804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:38.900606 804 log.cc:824] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:38.903875 804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: No bootstrap required, opened a new log
I20260928 02:52:38.912122 804 raft_consensus.cc:348] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } }
I20260928 02:52:38.912536 804 raft_consensus.cc:374] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:38.912616 804 raft_consensus.cc:729] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c201e316fcce4a8d8cebf3a466f6cc8f, State: Initialized, Role: FOLLOWER
I20260928 02:52:38.913241 804 consensus_queue.cc:260] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } }
I20260928 02:52:38.913465 804 raft_consensus.cc:388] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:38.913609 804 raft_consensus.cc:482] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:38.913789 804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:38.916489 804 raft_consensus.cc:504] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } }
I20260928 02:52:38.917006 804 leader_election.cc:302] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c201e316fcce4a8d8cebf3a466f6cc8f; no voters:
I20260928 02:52:38.917588 804 leader_election.cc:288] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:38.917750 809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:38.918478 809 raft_consensus.cc:686] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [term 1 LEADER]: Becoming Leader. State: Replica: c201e316fcce4a8d8cebf3a466f6cc8f, State: Running, Role: LEADER
I20260928 02:52:38.919214 809 consensus_queue.cc:237] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } }
I20260928 02:52:38.920212 804 sys_catalog.cc:564] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:38.922690 811 sys_catalog.cc:455] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c201e316fcce4a8d8cebf3a466f6cc8f. Latest consensus state: current_term: 1 leader_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } } }
I20260928 02:52:38.923125 811 sys_catalog.cc:458] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:38.923945 810 sys_catalog.cc:455] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c201e316fcce4a8d8cebf3a466f6cc8f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34133 } } }
I20260928 02:52:38.924482 810 sys_catalog.cc:458] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:38.926503 816 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:38.932991 816 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:38.942737 816 catalog_manager.cc:1347] Generated new cluster ID: 48341681de50491a893f8cdd8a2a3c59
I20260928 02:52:38.942929 816 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:38.956682 816 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:38.957180 816 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:38.964509 816 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: Generated new TSK 0
I20260928 02:52:38.965564 816 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:52:38.979120 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--builtin_ntp_servers=127.27.229.212:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 02:52:39.331547 828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:39.331841 828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:39.332034 828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:39.339607 828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:39.339722 828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:39.339910 828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:39.349761 828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:39.351298 828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:39.353283 828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:39.371227 834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:39.371690 836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:39.371315 833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:39.372253 828 server_base.cc:1034] running on GCE node
I20260928 02:52:39.372992 828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:39.374235 828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:39.375478 828 hybrid_clock.cc:648] HybridClock initialized: now 1790563959375416 us; error 53 us; skew 500 ppm
I20260928 02:52:39.375818 828 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:39.378180 828 webserver.cc:460] Webserver started at http://127.27.229.193:46873/ using document root <none> and password file <none>
I20260928 02:52:39.378994 828 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:39.379129 828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:39.379470 828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:39.382504 828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "c19c002956a74d5c9b0273ee4f27163c"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "c8368f9c188369f3cc4362d25790e49b"
server_key_iv: "5566eb178580eabeea7fd81ae9a2710f"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.383419 828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "c19c002956a74d5c9b0273ee4f27163c"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "c8368f9c188369f3cc4362d25790e49b"
server_key_iv: "5566eb178580eabeea7fd81ae9a2710f"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.388836 828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:39.392818 842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.394404 828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:39.394649 828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "c19c002956a74d5c9b0273ee4f27163c"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "c8368f9c188369f3cc4362d25790e49b"
server_key_iv: "5566eb178580eabeea7fd81ae9a2710f"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.394953 828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:39.401614 828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:39.402544 828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:39.402938 828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:39.403990 828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:39.405602 828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:39.405737 828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.405882 828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:39.405979 828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.442740 828 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:40565
I20260928 02:52:39.442799 954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:40565 every 8 connection(s)
I20260928 02:52:39.444204 828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:39.452611 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 828
I20260928 02:52:39.452852 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:39.453298 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:39.447091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:44277 (local address 127.27.229.254:34133)
0928 02:52:39.447581 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:39.447588 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:39.449004 (+ 1416us) server_negotiation.cc:407] Connection header received
0928 02:52:39.450598 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:39.450603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:39.450654 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:39.450756 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:39.452382 (+ 1626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:39.452389 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:39.452393 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:39.452415 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:39.452469 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:39.452487 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:39.452629 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:39.452909 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 02:52:39.452983 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":232,"threads_started":1}
I20260928 02:52:39.453682 28567 external_mini_cluster.cc:1428] Setting key e21ca5b632a943d9e66948f87dbaceb1
I20260928 02:52:39.454471 958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:39.447779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34133 (local address 127.27.229.193:44277)
0928 02:52:39.448736 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:39.448771 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:39.450323 (+ 1552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:39.451100 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:39.451109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:39.451208 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:39.452035 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:39.452042 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:39.452096 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:39.452190 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:39.452651 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:39.452663 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:39.452671 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:52:39.453013 (+ 342us) client_negotiation.cc:240] Negotiation successful
0928 02:52:39.453300 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":288,"threads_started":1}
I20260928 02:52:39.456205 955 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34133
I20260928 02:52:39.456533 955 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:39.456877 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--builtin_ntp_servers=127.27.229.212:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:52:39.457144 955 heartbeater.cc:511] Master 127.27.229.254:34133 requested a full tablet report, sending...
I20260928 02:52:39.459533 768 ts_manager.cc:194] Registered new tserver with Master: c19c002956a74d5c9b0273ee4f27163c (127.27.229.193:40565)
W20260928 02:52:39.798102 959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:39.798419 959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:39.798516 959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:39.806126 959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:39.806252 959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:39.806361 959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:39.816061 959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:39.817480 959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:39.819258 959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:39.833323 967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:39.833490 964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:39.833200 965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:39.834131 959 server_base.cc:1034] running on GCE node
I20260928 02:52:39.834760 959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:39.835844 959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:39.837133 959 hybrid_clock.cc:648] HybridClock initialized: now 1790563959837054 us; error 68 us; skew 500 ppm
I20260928 02:52:39.837504 959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:39.840422 959 webserver.cc:460] Webserver started at http://127.27.229.194:37557/ using document root <none> and password file <none>
I20260928 02:52:39.841216 959 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:39.841363 959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:39.841718 959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:39.844811 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "f3028278c3fc4fdc8afbdbc013c994d5"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "86e0ab2059116286d17ec7cce7e45d25"
server_key_iv: "b59b3e55936789b5010962ac9753ad96"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.845665 959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "f3028278c3fc4fdc8afbdbc013c994d5"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "86e0ab2059116286d17ec7cce7e45d25"
server_key_iv: "b59b3e55936789b5010962ac9753ad96"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.851333 959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:52:39.855897 973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.857713 959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:52:39.857930 959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "f3028278c3fc4fdc8afbdbc013c994d5"
format_stamp: "Formatted at 2026-09-28 02:52:39 on dist-test-slave-s1kg"
server_key: "86e0ab2059116286d17ec7cce7e45d25"
server_key_iv: "b59b3e55936789b5010962ac9753ad96"
server_key_version: "encryptionkey@0"
I20260928 02:52:39.858182 959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:39.864311 959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:39.865211 959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:39.865504 959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:39.866459 959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:39.868039 959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:39.868160 959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.868301 959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:39.868391 959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:39.904220 959 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41809
I20260928 02:52:39.904278 1085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41809 every 8 connection(s)
I20260928 02:52:39.905587 959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:39.908757 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 959
I20260928 02:52:39.908921 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:39.909389 28567 external_mini_cluster.cc:1428] Setting key acca810a733b48acfb54ede6cdce770f
I20260928 02:52:39.912503 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--builtin_ntp_servers=127.27.229.212:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 02:52:39.915120 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:39.908256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37757 (local address 127.27.229.254:34133)
0928 02:52:39.908473 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:39.908479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:39.910112 (+ 1633us) server_negotiation.cc:407] Connection header received
0928 02:52:39.912196 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:39.912202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:39.912265 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:39.912388 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:39.914267 (+ 1879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:39.914274 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:39.914278 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:39.914301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:39.914339 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:39.914351 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:39.914513 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:39.914753 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:52:39.914823 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:52:39.916160 1088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:39.908777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34133 (local address 127.27.229.194:37757)
0928 02:52:39.909939 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:39.909984 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:39.911911 (+ 1927us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:39.912657 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:39.912670 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:39.913041 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:39.913877 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:39.913879 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:39.913951 (+ 72us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:39.914046 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:39.914475 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:39.914487 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:39.914496 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:52:39.914877 (+ 381us) client_negotiation.cc:240] Negotiation successful
0928 02:52:39.915180 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":389,"threads_started":1}
I20260928 02:52:39.917747 1086 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34133
I20260928 02:52:39.918140 1086 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:39.918987 1086 heartbeater.cc:511] Master 127.27.229.254:34133 requested a full tablet report, sending...
I20260928 02:52:39.921020 768 ts_manager.cc:194] Registered new tserver with Master: f3028278c3fc4fdc8afbdbc013c994d5 (127.27.229.194:41809)
W20260928 02:52:40.248039 1089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:40.248340 1089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:40.248492 1089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:40.256037 1089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:40.256150 1089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:40.256325 1089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:40.265713 1089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:40.267089 1089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:40.268887 1089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:40.282403 1094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:40.282536 1095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:40.282483 1097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:40.283581 1089 server_base.cc:1034] running on GCE node
I20260928 02:52:40.284281 1089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:40.285245 1089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:40.286484 1089 hybrid_clock.cc:648] HybridClock initialized: now 1790563960286423 us; error 56 us; skew 500 ppm
I20260928 02:52:40.286823 1089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:40.289326 1089 webserver.cc:460] Webserver started at http://127.27.229.195:42507/ using document root <none> and password file <none>
I20260928 02:52:40.290076 1089 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:40.290216 1089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:40.290562 1089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:40.293582 1089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32"
format_stamp: "Formatted at 2026-09-28 02:52:40 on dist-test-slave-s1kg"
server_key: "ecffd239f0898002f884cd1055434c4c"
server_key_iv: "1da0e99609d77f795d8c6d4de757331f"
server_key_version: "encryptionkey@0"
I20260928 02:52:40.294431 1089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32"
format_stamp: "Formatted at 2026-09-28 02:52:40 on dist-test-slave-s1kg"
server_key: "ecffd239f0898002f884cd1055434c4c"
server_key_iv: "1da0e99609d77f795d8c6d4de757331f"
server_key_version: "encryptionkey@0"
I20260928 02:52:40.300122 1089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:52:40.303933 1103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:40.305431 1089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:40.305714 1089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32"
format_stamp: "Formatted at 2026-09-28 02:52:40 on dist-test-slave-s1kg"
server_key: "ecffd239f0898002f884cd1055434c4c"
server_key_iv: "1da0e99609d77f795d8c6d4de757331f"
server_key_version: "encryptionkey@0"
I20260928 02:52:40.305972 1089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:40.311820 1089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:40.312811 1089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:40.313091 1089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:40.313992 1089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:40.315505 1089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:40.315621 1089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:40.315785 1089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:40.315851 1089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:40.349699 1089 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45011
I20260928 02:52:40.349756 1215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45011 every 8 connection(s)
I20260928 02:52:40.351171 1089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:40.355324 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1089
I20260928 02:52:40.355494 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:40.356045 28567 external_mini_cluster.cc:1428] Setting key c6d5f813daa3aa28d2aee73a7f696666
I20260928 02:52:40.360399 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.353785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:54999 (local address 127.27.229.254:34133)
0928 02:52:40.353968 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.353991 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.355700 (+ 1709us) server_negotiation.cc:407] Connection header received
0928 02:52:40.358017 (+ 2317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.358022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.358090 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.358208 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.359666 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.359672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.359677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.359701 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.359732 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.359743 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.359887 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.360077 (+ 190us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.360144 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:52:40.361408 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.354338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34133 (local address 127.27.229.195:54999)
0928 02:52:40.355514 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.355566 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.357707 (+ 2141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.358451 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.358460 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.358553 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.359366 (+ 813us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.359369 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.359407 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.359494 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.359840 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.359855 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.359865 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.360155 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.360439 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":291,"threads_started":1}
I20260928 02:52:40.362883 1216 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34133
I20260928 02:52:40.363236 1216 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:40.364075 1216 heartbeater.cc:511] Master 127.27.229.254:34133 requested a full tablet report, sending...
I20260928 02:52:40.365437 768 ts_manager.cc:194] Registered new tserver with Master: e71ce70bdecb4d6d9bc3fb7405c1ee32 (127.27.229.195:45011)
I20260928 02:52:40.371860 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:52:40.380053 957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.377025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50854 (local address 127.27.229.254:34133)
0928 02:52:40.377285 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.377291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.378009 (+ 718us) server_negotiation.cc:407] Connection header received
0928 02:52:40.378214 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.378220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.378282 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.378395 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.379400 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.379413 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.379417 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.379433 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.379465 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.379476 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.379623 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.379720 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.379783 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:52:40.384434 768 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:40.391460 768 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:40.425329 1232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.403957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41809 (local address 127.0.0.1:46950)
0928 02:52:40.412900 (+ 8943us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.412971 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.413184 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.420760 (+ 7576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.420769 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.422615 (+ 1846us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.423825 (+ 1210us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.423829 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.423863 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.423956 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.424530 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.424545 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.424555 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.424712 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.424798 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8512,"thread_start_us":19633,"threads_started":1}
I20260928 02:52:40.426687 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.404045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.27.229.194:41809)
0928 02:52:40.419925 (+ 15880us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.419940 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.419993 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:52:40.420109 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.420123 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.420514 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.420724 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.424226 (+ 3502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.424237 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.424252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.424274 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.424341 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.424373 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.426097 (+ 1724us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.426242 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.426354 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15620,"thread_start_us":18625,"threads_started":1}
I20260928 02:52:40.427124 1230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.404127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45011 (local address 127.0.0.1:60292)
0928 02:52:40.422910 (+ 18783us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.422943 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.423316 (+ 373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.425125 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.425136 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.425161 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.425945 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.425947 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.425989 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.426040 (+ 51us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.426464 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.426468 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.426473 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.426601 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.426656 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18448,"thread_start_us":12124,"threads_started":1}
I20260928 02:52:40.427160 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.404384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.27.229.195:45011)
0928 02:52:40.422623 (+ 18239us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.422636 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.423460 (+ 824us) server_negotiation.cc:407] Connection header received
0928 02:52:40.423560 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.423570 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.423825 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.423999 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.426230 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.426238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.426253 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.426273 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.426334 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.426354 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.426528 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.426599 (+ 71us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.426693 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18012,"thread_start_us":17791,"threads_started":1}
I20260928 02:52:40.428731 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.407343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51688 (local address 127.27.229.193:40565)
0928 02:52:40.423411 (+ 16068us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.423423 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.424125 (+ 702us) server_negotiation.cc:407] Connection header received
0928 02:52:40.424198 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.424205 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.424402 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.424536 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.426387 (+ 1851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.426393 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.426402 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.426451 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.426513 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.426566 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.426688 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.428231 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.428371 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15865,"thread_start_us":15716,"threads_started":1}
I20260928 02:52:40.429145 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.406677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51688)
0928 02:52:40.423873 (+ 17196us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.423890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.423993 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.425302 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.425306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.425329 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.426185 (+ 856us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.426188 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.426202 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.426240 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.426736 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.426743 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.426750 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.427532 (+ 782us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.427585 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17067,"thread_start_us":5565,"threads_started":1}
I20260928 02:52:40.431123 1021 tablet_service.cc:1467] Processing CreateTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 (DEFAULT_TABLE table=test-table [id=391cc3ec1b9447adbf7248f2ba4027ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:40.431342 1151 tablet_service.cc:1467] Processing CreateTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 (DEFAULT_TABLE table=test-table [id=391cc3ec1b9447adbf7248f2ba4027ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:40.432673 1151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2811962b9244400fa2ca9da3a1ff4dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:40.432729 1021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2811962b9244400fa2ca9da3a1ff4dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:40.432688 890 tablet_service.cc:1467] Processing CreateTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 (DEFAULT_TABLE table=test-table [id=391cc3ec1b9447adbf7248f2ba4027ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:40.433959 890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2811962b9244400fa2ca9da3a1ff4dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:40.450141 1235 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Bootstrap starting.
I20260928 02:52:40.451303 1234 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Bootstrap starting.
I20260928 02:52:40.454077 1236 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: Bootstrap starting.
I20260928 02:52:40.455452 1235 tablet_bootstrap.cc:654] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:40.455457 1234 tablet_bootstrap.cc:654] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:40.456751 1234 log.cc:824] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:40.456751 1235 log.cc:824] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:40.457479 1236 tablet_bootstrap.cc:654] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:40.458774 1236 log.cc:824] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:40.461221 1234 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: No bootstrap required, opened a new log
I20260928 02:52:40.461488 1235 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: No bootstrap required, opened a new log
I20260928 02:52:40.461498 1234 ts_tablet_manager.cc:1397] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260928 02:52:40.461750 1236 tablet_bootstrap.cc:492] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: No bootstrap required, opened a new log
I20260928 02:52:40.461781 1235 ts_tablet_manager.cc:1397] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260928 02:52:40.462007 1236 ts_tablet_manager.cc:1397] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:52:40.462819 955 heartbeater.cc:503] Master 127.27.229.254:34133 was elected leader, sending a full tablet report...
I20260928 02:52:40.471226 1234 raft_consensus.cc:348] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.471639 1234 raft_consensus.cc:374] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:40.471777 1234 raft_consensus.cc:729] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e71ce70bdecb4d6d9bc3fb7405c1ee32, State: Initialized, Role: FOLLOWER
I20260928 02:52:40.472416 1234 consensus_queue.cc:260] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.472693 1235 raft_consensus.cc:348] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.472692 1236 raft_consensus.cc:348] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.473192 1236 raft_consensus.cc:374] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:40.473196 1235 raft_consensus.cc:374] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:40.473337 1236 raft_consensus.cc:729] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c19c002956a74d5c9b0273ee4f27163c, State: Initialized, Role: FOLLOWER
I20260928 02:52:40.473393 1235 raft_consensus.cc:729] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3028278c3fc4fdc8afbdbc013c994d5, State: Initialized, Role: FOLLOWER
I20260928 02:52:40.473609 1216 heartbeater.cc:503] Master 127.27.229.254:34133 was elected leader, sending a full tablet report...
I20260928 02:52:40.474041 1235 consensus_queue.cc:260] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.474078 1236 consensus_queue.cc:260] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.474313 1234 ts_tablet_manager.cc:1428] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260928 02:52:40.478267 1086 heartbeater.cc:503] Master 127.27.229.254:34133 was elected leader, sending a full tablet report...
I20260928 02:52:40.479497 1235 ts_tablet_manager.cc:1428] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Time spent starting tablet: real 0.018s user 0.008s sys 0.008s
I20260928 02:52:40.480162 1236 ts_tablet_manager.cc:1428] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260928 02:52:40.531433 1240 raft_consensus.cc:482] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:40.531775 1240 raft_consensus.cc:504] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.533579 1240 leader_election.cc:288] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3028278c3fc4fdc8afbdbc013c994d5 (127.27.229.194:41809), c19c002956a74d5c9b0273ee4f27163c (127.27.229.193:40565)
I20260928 02:52:40.536499 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.533962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:51983 (local address 127.27.229.194:41809)
0928 02:52:40.534144 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.534150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.534172 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:52:40.534430 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.534435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.534499 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.534598 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.535783 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.535792 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.535799 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.535821 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.535863 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.535883 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.536016 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.536154 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.536203 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:40.536521 1218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.533816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41809 (local address 127.27.229.195:51983)
0928 02:52:40.534037 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.534061 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.534180 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.534756 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.534760 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.534778 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.535496 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.535499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.535513 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.535575 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.535998 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.536002 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.536006 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.536140 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.536196 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:52:40.536794 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.534432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:43335 (local address 127.27.229.193:40565)
0928 02:52:40.534609 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.534615 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.534820 (+ 205us) server_negotiation.cc:407] Connection header received
0928 02:52:40.535046 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.535050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.535096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.535202 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.536153 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.536160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.536164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.536183 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.536218 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.536232 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.536347 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.536474 (+ 127us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.536536 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:52:40.537109 1243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.533938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.27.229.195:43335)
0928 02:52:40.534714 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.534728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:40.534892 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:40.535219 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:40.535223 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:40.535247 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:40.535929 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:52:40.535935 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:40.535952 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:52:40.535994 (+ 42us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:40.536340 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:40.536345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:40.536350 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:52:40.536475 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 02:52:40.536533 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":464,"threads_started":1}
I20260928 02:52:40.537679 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2811962b9244400fa2ca9da3a1ff4dd1" candidate_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" is_pre_election: true
I20260928 02:52:40.538010 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2811962b9244400fa2ca9da3a1ff4dd1" candidate_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19c002956a74d5c9b0273ee4f27163c" is_pre_election: true
I20260928 02:52:40.538193 1041 raft_consensus.cc:2454] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e71ce70bdecb4d6d9bc3fb7405c1ee32 in term 0.
I20260928 02:52:40.538512 910 raft_consensus.cc:2454] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e71ce70bdecb4d6d9bc3fb7405c1ee32 in term 0.
I20260928 02:52:40.539002 1104 leader_election.cc:302] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e71ce70bdecb4d6d9bc3fb7405c1ee32, f3028278c3fc4fdc8afbdbc013c994d5; no voters:
I20260928 02:52:40.539467 1240 raft_consensus.cc:2781] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:40.539644 1240 raft_consensus.cc:482] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:40.539772 1240 raft_consensus.cc:3037] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:40.542003 1240 raft_consensus.cc:504] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.542659 1240 leader_election.cc:288] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [CANDIDATE]: Term 1 election: Requested vote from peers f3028278c3fc4fdc8afbdbc013c994d5 (127.27.229.194:41809), c19c002956a74d5c9b0273ee4f27163c (127.27.229.193:40565)
I20260928 02:52:40.543366 910 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2811962b9244400fa2ca9da3a1ff4dd1" candidate_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c19c002956a74d5c9b0273ee4f27163c"
I20260928 02:52:40.543367 1041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2811962b9244400fa2ca9da3a1ff4dd1" candidate_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3028278c3fc4fdc8afbdbc013c994d5"
I20260928 02:52:40.543606 1041 raft_consensus.cc:3037] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:40.543606 910 raft_consensus.cc:3037] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:40.547039 910 raft_consensus.cc:2454] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e71ce70bdecb4d6d9bc3fb7405c1ee32 in term 1.
I20260928 02:52:40.547047 1041 raft_consensus.cc:2454] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e71ce70bdecb4d6d9bc3fb7405c1ee32 in term 1.
I20260928 02:52:40.547636 1107 leader_election.cc:302] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c19c002956a74d5c9b0273ee4f27163c, e71ce70bdecb4d6d9bc3fb7405c1ee32, f3028278c3fc4fdc8afbdbc013c994d5; no voters:
I20260928 02:52:40.548022 1240 raft_consensus.cc:2781] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:40.548429 1240 raft_consensus.cc:686] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 1 LEADER]: Becoming Leader. State: Replica: e71ce70bdecb4d6d9bc3fb7405c1ee32, State: Running, Role: LEADER
I20260928 02:52:40.548913 1240 consensus_queue.cc:237] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } }
I20260928 02:52:40.553697 767 catalog_manager.cc:5495] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 reported cstate change: term changed from 0 to 1, leader changed from <none> to e71ce70bdecb4d6d9bc3fb7405c1ee32 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e71ce70bdecb4d6d9bc3fb7405c1ee32" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45011 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:40.594619 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.592378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.27.229.195:45011)
0928 02:52:40.592604 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.592609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.592627 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:52:40.592797 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.592801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.592902 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.592989 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.593926 (+ 937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.593931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.593934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.593947 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.593977 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.593989 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.594091 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.594261 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.594354 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:52:40.602727 1217 tablet.cc:2369] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:40.603438 1041 raft_consensus.cc:1261] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 1 FOLLOWER]: Refusing update from remote peer e71ce70bdecb4d6d9bc3fb7405c1ee32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:40.603438 910 raft_consensus.cc:1261] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 1 FOLLOWER]: Refusing update from remote peer e71ce70bdecb4d6d9bc3fb7405c1ee32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:40.604599 1244 consensus_queue.cc:1035] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c19c002956a74d5c9b0273ee4f27163c" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:40.604950 1240 consensus_queue.cc:1035] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3028278c3fc4fdc8afbdbc013c994d5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:40.620713 1247 mvcc.cc:205] Tried to move back new op lower bound from 7334149982620405760 to 7334149982412251136. Current Snapshot: MvccSnapshot[applied={T|T < 7334149982620405760}]
I20260928 02:52:40.622068 1249 mvcc.cc:205] Tried to move back new op lower bound from 7334149982620405760 to 7334149982412251136. Current Snapshot: MvccSnapshot[applied={T|T < 7334149982620405760}]
I20260928 02:52:40.640786 767 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:40.641162 767 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:40.645753 767 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c201e316fcce4a8d8cebf3a466f6cc8f: Sending DeleteTablet for 3 replicas of tablet 2811962b9244400fa2ca9da3a1ff4dd1
I20260928 02:52:40.646878 1151 tablet_service.cc:1514] Processing DeleteTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:40 UTC) from {username='slave'} at 127.0.0.1:60292
I20260928 02:52:40.646941 1021 tablet_service.cc:1514] Processing DeleteTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:40 UTC) from {username='slave'} at 127.0.0.1:46950
I20260928 02:52:40.647828 1258 tablet_replica.cc:330] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: stopping tablet replica
I20260928 02:52:40.648231 890 tablet_service.cc:1514] Processing DeleteTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:40 UTC) from {username='slave'} at 127.0.0.1:51688
I20260928 02:52:40.648308 1258 raft_consensus.cc:2229] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:40.648342 889 tablet_service.cc:1514] Processing DeleteTablet for tablet 2811962b9244400fa2ca9da3a1ff4dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:40 UTC) from {username='slave'} at 127.0.0.1:51688
I20260928 02:52:40.648836 1258 raft_consensus.cc:2258] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:40.649645 1257 tablet_replica.cc:330] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: stopping tablet replica
I20260928 02:52:40.649817 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 828
I20260928 02:52:40.650128 1257 raft_consensus.cc:2229] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:40.650780 1257 raft_consensus.cc:2258] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:40.655351 1259 tablet_replica.cc:330] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c: stopping tablet replica
I20260928 02:52:40.655771 1257 ts_tablet_manager.cc:1905] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:40.656360 1233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.651245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51704 (local address 127.27.229.193:40565)
0928 02:52:40.651464 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.651473 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.653841 (+ 2368us) server_negotiation.cc:407] Connection header received
0928 02:52:40.654144 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.654153 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.654274 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.654353 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.655590 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.655596 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.655601 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.655620 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.655658 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.655673 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.655770 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.656030 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.656090 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:52:40.656886 1258 ts_tablet_manager.cc:1905] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:40.657291 950 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:40.664472 1258 ts_tablet_manager.cc:1918] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:40.664758 1258 log.cc:1196] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/2811962b9244400fa2ca9da3a1ff4dd1
I20260928 02:52:40.665388 1258 ts_tablet_manager.cc:1939] T 2811962b9244400fa2ca9da3a1ff4dd1 P f3028278c3fc4fdc8afbdbc013c994d5: Deleting consensus metadata
I20260928 02:52:40.667186 1257 ts_tablet_manager.cc:1918] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:40.667445 1257 log.cc:1196] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/2811962b9244400fa2ca9da3a1ff4dd1
I20260928 02:52:40.667510 753 catalog_manager.cc:4841] TS f3028278c3fc4fdc8afbdbc013c994d5 (127.27.229.194:41809): tablet 2811962b9244400fa2ca9da3a1ff4dd1 (table test-table [id=391cc3ec1b9447adbf7248f2ba4027ac]) successfully deleted
I20260928 02:52:40.668134 1257 ts_tablet_manager.cc:1939] T 2811962b9244400fa2ca9da3a1ff4dd1 P e71ce70bdecb4d6d9bc3fb7405c1ee32: Deleting consensus metadata
I20260928 02:52:40.670714 755 catalog_manager.cc:4841] TS e71ce70bdecb4d6d9bc3fb7405c1ee32 (127.27.229.195:45011): tablet 2811962b9244400fa2ca9da3a1ff4dd1 (table test-table [id=391cc3ec1b9447adbf7248f2ba4027ac]) successfully deleted
I20260928 02:52:40.657569 1259 raft_consensus.cc:2229] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:40.756289 1259 raft_consensus.cc:2258] T 2811962b9244400fa2ca9da3a1ff4dd1 P c19c002956a74d5c9b0273ee4f27163c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:40.756466 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 828
I20260928 02:52:40.774504 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 959
W20260928 02:52:40.774602 756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.229.193:40565 (error 108)
I20260928 02:52:40.777102 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.776619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51718)
0928 02:52:40.776805 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.776964 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 02:52:40.777454 756 catalog_manager.cc:4584] TS c19c002956a74d5c9b0273ee4f27163c (127.27.229.193:40565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2811962b9244400fa2ca9da3a1ff4dd1: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
I20260928 02:52:40.777985 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.775786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46964 (local address 127.27.229.194:41809)
0928 02:52:40.775967 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.775974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.775997 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:40.776064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.776070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.776173 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.776289 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.777304 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.777310 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.777314 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.777332 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.777368 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.777383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.777513 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.777651 (+ 138us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.777709 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:52:40.778622 1081 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:40.822163 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.821632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51734)
0928 02:52:40.821855 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.821968 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 02:52:40.823443 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.822974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51738)
0928 02:52:40.823215 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.823303 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260928 02:52:40.830906 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.830344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51740)
0928 02:52:40.830562 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.830682 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:52:40.832077 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.831658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51756)
0928 02:52:40.831831 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.831921 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:52:40.868499 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 959
I20260928 02:52:40.871805 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.871235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51758)
0928 02:52:40.871492 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.871596 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260928 02:52:40.872972 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.872581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51764)
0928 02:52:40.872797 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.872861 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 02:52:40.890766 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1089
I20260928 02:52:40.894024 1231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.891722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60318 (local address 127.27.229.195:45011)
0928 02:52:40.891939 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:40.891944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:40.891967 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:40.892048 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:40.892052 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:40.892097 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:40.892200 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:40.893251 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:40.893257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:40.893262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:40.893286 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:40.893317 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:40.893328 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:40.893520 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:40.893616 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 02:52:40.893677 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:52:40.894800 1211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:40.909492 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.909008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51774)
0928 02:52:40.909238 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.909325 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260928 02:52:40.910463 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.910149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51790)
0928 02:52:40.910291 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.910362 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:52:40.974170 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.973677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51804)
0928 02:52:40.973924 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.974013 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260928 02:52:40.975348 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:40.974986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51806)
0928 02:52:40.975141 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:40.975227 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:52:41.012012 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:41.011283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51818)
0928 02:52:41.011642 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:41.011759 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
I20260928 02:52:41.013443 1229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:41.013104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40565 (local address 127.0.0.1:51830)
0928 02:52:41.013239 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:41.013320 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 02:52:41.013511 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1089
I20260928 02:52:41.032780 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 738
I20260928 02:52:41.034040 799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:41.140595 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 738
2026-09-28T02:52:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2751 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 02:52:41.182533 28567 mini_oidc.cc:152] Starting JWKS server
I20260928 02:52:41.185279 28567 webserver.cc:460] Webserver started at https://127.0.0.1:43125/ using document root <none> and password file <none>
I20260928 02:52:41.185612 28567 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:52:41.186348 28567 webserver.cc:460] Webserver started at http://127.0.0.1:46073/ using document root <none> and password file <none>
2026-09-28T02:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:41Z Disabled control of system clock
I20260928 02:52:41.215663 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34095
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:35087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34095
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43125/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/testchainca.cert with env {}
W20260928 02:52:41.554069 1271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:41.554370 1271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:41.554458 1271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:41.562309 1271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:41.562439 1271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:41.562479 1271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:41.562527 1271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:41.562572 1271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:52:41.562633 1271 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:52:41.562674 1271 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43125/jwks.json
I20260928 02:52:41.573642 1271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34095
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43125/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:41.575111 1271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:41.576372 1271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:41.588654 1276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:41.588620 1277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:41.588589 1279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:41.589775 1271 server_base.cc:1034] running on GCE node
I20260928 02:52:41.590669 1271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:41.592254 1271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:41.593501 1271 hybrid_clock.cc:648] HybridClock initialized: now 1790563961593437 us; error 61 us; skew 500 ppm
I20260928 02:52:41.593976 1271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:41.596478 1271 webserver.cc:460] Webserver started at http://127.27.229.254:39963/ using document root <none> and password file <none>
I20260928 02:52:41.597223 1271 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:41.597352 1271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:41.597687 1271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:41.600847 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "995857ea62794eadbc3ffdb984ea2ea5"
format_stamp: "Formatted at 2026-09-28 02:52:41 on dist-test-slave-s1kg"
server_key: "b2a12c1086434c8d20d3e68f07a134b1"
server_key_iv: "a8f980d35a99e52aa1fcc79a0970fa11"
server_key_version: "encryptionkey@0"
I20260928 02:52:41.601681 1271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "995857ea62794eadbc3ffdb984ea2ea5"
format_stamp: "Formatted at 2026-09-28 02:52:41 on dist-test-slave-s1kg"
server_key: "b2a12c1086434c8d20d3e68f07a134b1"
server_key_iv: "a8f980d35a99e52aa1fcc79a0970fa11"
server_key_version: "encryptionkey@0"
I20260928 02:52:41.607229 1271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:52:41.610901 1285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:41.612380 1271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:41.612589 1271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "995857ea62794eadbc3ffdb984ea2ea5"
format_stamp: "Formatted at 2026-09-28 02:52:41 on dist-test-slave-s1kg"
server_key: "b2a12c1086434c8d20d3e68f07a134b1"
server_key_iv: "a8f980d35a99e52aa1fcc79a0970fa11"
server_key_version: "encryptionkey@0"
I20260928 02:52:41.612824 1271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:41.635335 1271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:41.636264 1271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:41.636610 1271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:41.656615 1271 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34095
I20260928 02:52:41.656638 1336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34095 every 8 connection(s)
I20260928 02:52:41.658038 1271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:41.662212 1337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:41.666711 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1271
I20260928 02:52:41.667039 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:41.667682 28567 external_mini_cluster.cc:1428] Setting key 988b063aac6966a70af9cca52d8b1e9b
I20260928 02:52:41.674319 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5: Bootstrap starting.
I20260928 02:52:41.677578 1337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:41.678684 1337 log.cc:824] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:41.682338 1337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5: No bootstrap required, opened a new log
I20260928 02:52:41.682593 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:41.669778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38384 (local address 127.27.229.254:34095)
0928 02:52:41.671113 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:41.671126 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:41.671178 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:52:41.672947 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:41.672973 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:41.673627 (+ 654us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:41.674101 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:41.674656 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.676029 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.677450 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.678218 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.678435 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:41.679801 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:41.679809 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:41.679824 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:41.680532 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:41.680634 (+ 102us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:41.680660 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:41.680779 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:41.681004 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:52:41.681335 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":424,"threads_started":1}
I20260928 02:52:41.686730 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:52:41.691454 1337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } }
I20260928 02:52:41.691815 1337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:41.691890 1337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995857ea62794eadbc3ffdb984ea2ea5, State: Initialized, Role: FOLLOWER
I20260928 02:52:41.692401 1337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } }
I20260928 02:52:41.692647 1337 raft_consensus.cc:388] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:41.692809 1337 raft_consensus.cc:482] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:41.693013 1337 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:41.695819 1337 raft_consensus.cc:504] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } }
I20260928 02:52:41.696424 1337 leader_election.cc:302] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 995857ea62794eadbc3ffdb984ea2ea5; no voters:
I20260928 02:52:41.697129 1337 leader_election.cc:288] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:41.697239 1342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:41.698058 1342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [term 1 LEADER]: Becoming Leader. State: Replica: 995857ea62794eadbc3ffdb984ea2ea5, State: Running, Role: LEADER
I20260928 02:52:41.698625 1342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } }
I20260928 02:52:41.699592 1337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:41.701481 1343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "995857ea62794eadbc3ffdb984ea2ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } } }
I20260928 02:52:41.702229 1343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:41.701495 1344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 995857ea62794eadbc3ffdb984ea2ea5. Latest consensus state: current_term: 1 leader_uuid: "995857ea62794eadbc3ffdb984ea2ea5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "995857ea62794eadbc3ffdb984ea2ea5" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34095 } } }
I20260928 02:52:41.702843 1344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:41.705103 1349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:41.709949 1349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:41.718556 1349 catalog_manager.cc:1347] Generated new cluster ID: 699c6990ac394c7da824167a20352c48
I20260928 02:52:41.718679 1349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:41.733235 1349 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:41.734557 1349 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:41.739986 1349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 995857ea62794eadbc3ffdb984ea2ea5: Generated new TSK 0
I20260928 02:52:41.741032 1349 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:52:41.790479 1367 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 02:52:41.790670 1367 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:52:41.791859 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:41.769829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38390 (local address 127.27.229.254:34095)
0928 02:52:41.770147 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:41.770156 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:41.770798 (+ 642us) server_negotiation.cc:407] Connection header received
0928 02:52:41.771188 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:41.771192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:41.771273 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:41.771428 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:52:41.772014 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.772737 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.774198 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.774659 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.774805 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:41.775398 (+ 593us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:52:41.791157 (+ 15759us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:52:41.791558 (+ 401us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":120}
W20260928 02:52:41.791992 1340 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38390: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 02:52:41.793295 1366 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:41.769681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34095 (local address 127.0.0.1:38390)
0928 02:52:41.770682 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:41.770699 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:41.770949 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:41.771477 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:41.771482 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:41.771496 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:52:41.771842 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:41.771850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.772883 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.772889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:41.774040 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:41.774051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:41.774890 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:41.774899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:41.775037 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:41.775104 (+ 67us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:52:41.791471 (+ 16367us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 02:52:41.791833 (+ 362us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.229.254:34095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":811,"thread_start_us":658,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.229.254:34095: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43125/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 02:52:41.795728 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1271
I20260928 02:52:41.797117 1332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:41.891775 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1271
2026-09-28T02:52:41Z chronyd exiting
I20260928 02:52:41.942986 28567 test_util.cc:175] -----------------------------------------------
I20260928 02:52:41.943198 28567 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790563915649057-28567-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (765 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 02:52:41.947098 28567 mini_oidc.cc:152] Starting JWKS server
I20260928 02:52:41.948940 28567 webserver.cc:460] Webserver started at https://127.0.0.1:36963/ using document root <none> and password file <none>
I20260928 02:52:41.949204 28567 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:52:41.949916 28567 webserver.cc:460] Webserver started at http://127.0.0.1:45625/ using document root <none> and password file <none>
2026-09-28T02:52:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:41Z Disabled control of system clock
I20260928 02:52:41.978513 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41273
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:32979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:41273
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36963/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/testchainca.cert with env {}
W20260928 02:52:42.307885 1374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:42.308200 1374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:42.308287 1374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:42.315516 1374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:42.315639 1374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:42.315675 1374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:42.315725 1374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:42.315765 1374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:52:42.315973 1374 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:52:42.316092 1374 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36963/jwks.json
I20260928 02:52:42.326502 1374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:32979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:41273
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36963/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:42.327885 1374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:42.329123 1374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:42.341356 1379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:42.341308 1382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:42.341267 1380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:42.342489 1374 server_base.cc:1034] running on GCE node
I20260928 02:52:42.343432 1374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:42.344956 1374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:42.346228 1374 hybrid_clock.cc:648] HybridClock initialized: now 1790563962346169 us; error 53 us; skew 500 ppm
I20260928 02:52:42.346670 1374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:42.352947 1374 webserver.cc:460] Webserver started at http://127.27.229.254:41701/ using document root <none> and password file <none>
I20260928 02:52:42.353736 1374 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:42.353878 1374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:42.354221 1374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:42.357282 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "e4cba97c6e8a49708e5efb632d275488"
format_stamp: "Formatted at 2026-09-28 02:52:42 on dist-test-slave-s1kg"
server_key: "c4c95ce19645eb3dd5703c4332a3944a"
server_key_iv: "28072a8d412242889e6abc81daef320c"
server_key_version: "encryptionkey@0"
I20260928 02:52:42.358111 1374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "e4cba97c6e8a49708e5efb632d275488"
format_stamp: "Formatted at 2026-09-28 02:52:42 on dist-test-slave-s1kg"
server_key: "c4c95ce19645eb3dd5703c4332a3944a"
server_key_iv: "28072a8d412242889e6abc81daef320c"
server_key_version: "encryptionkey@0"
I20260928 02:52:42.363464 1374 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:52:42.367081 1388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:42.368677 1374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:42.368911 1374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "e4cba97c6e8a49708e5efb632d275488"
format_stamp: "Formatted at 2026-09-28 02:52:42 on dist-test-slave-s1kg"
server_key: "c4c95ce19645eb3dd5703c4332a3944a"
server_key_iv: "28072a8d412242889e6abc81daef320c"
server_key_version: "encryptionkey@0"
I20260928 02:52:42.369163 1374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:42.388373 1374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:42.389261 1374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:42.389590 1374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:42.408991 1374 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:41273
I20260928 02:52:42.409030 1439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:41273 every 8 connection(s)
I20260928 02:52:42.410420 1374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:42.414376 1440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:42.419826 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1374
I20260928 02:52:42.420032 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:42.420800 28567 external_mini_cluster.cc:1428] Setting key eee376cbbc6fc117ff5a16691889be60
I20260928 02:52:42.426245 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488: Bootstrap starting.
I20260928 02:52:42.429382 1440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:42.430431 1440 log.cc:824] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:42.433507 1440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488: No bootstrap required, opened a new log
I20260928 02:52:42.434525 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:42.422963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.27.229.254:41273)
0928 02:52:42.424005 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:42.424019 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:42.424066 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:52:42.425729 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:42.425752 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:42.426251 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:42.426699 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:42.427286 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.428566 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.429898 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.430660 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.430891 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:42.432184 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:42.432192 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:42.432207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:42.432767 (+ 560us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:42.432843 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:42.432864 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:42.433001 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:42.433254 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 02:52:42.433585 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":334,"threads_started":1}
I20260928 02:52:42.438364 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:52:42.441565 1440 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } }
I20260928 02:52:42.442031 1440 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:42.442180 1440 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cba97c6e8a49708e5efb632d275488, State: Initialized, Role: FOLLOWER
I20260928 02:52:42.442745 1440 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } }
I20260928 02:52:42.443084 1440 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:42.443236 1440 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:42.443413 1440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:42.445581 1440 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } }
I20260928 02:52:42.446141 1440 leader_election.cc:302] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cba97c6e8a49708e5efb632d275488; no voters:
I20260928 02:52:42.446770 1440 leader_election.cc:288] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:42.446990 1445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:42.448230 1445 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [term 1 LEADER]: Becoming Leader. State: Replica: e4cba97c6e8a49708e5efb632d275488, State: Running, Role: LEADER
I20260928 02:52:42.448758 1445 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } }
I20260928 02:52:42.449096 1440 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:42.451565 1447 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cba97c6e8a49708e5efb632d275488. Latest consensus state: current_term: 1 leader_uuid: "e4cba97c6e8a49708e5efb632d275488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } } }
I20260928 02:52:42.451984 1447 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:42.451665 1446 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4cba97c6e8a49708e5efb632d275488" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cba97c6e8a49708e5efb632d275488" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41273 } } }
I20260928 02:52:42.452490 1446 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:42.452841 1451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:42.458959 1451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:42.467962 1451 catalog_manager.cc:1347] Generated new cluster ID: 10f8fc9f3c9e43878efdd50b15174b08
I20260928 02:52:42.468107 1451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:42.487495 1451 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:42.488883 1451 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:42.499197 1451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4cba97c6e8a49708e5efb632d275488: Generated new TSK 0
I20260928 02:52:42.500098 1451 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 02:52:42.540231 1470 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 02:52:42.540374 1470 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:52:42.541638 1443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:42.520420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46070 (local address 127.27.229.254:41273)
0928 02:52:42.520620 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:42.520627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:42.521000 (+ 373us) server_negotiation.cc:407] Connection header received
0928 02:52:42.521265 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:42.521272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:42.521367 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:42.521551 (+ 184us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:52:42.522022 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.523236 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.524744 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.525477 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.525612 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:42.526158 (+ 546us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:52:42.540927 (+ 14769us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:52:42.541358 (+ 431us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46070: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":74}
W20260928 02:52:42.541744 1469 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:42.520237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41273 (local address 127.0.0.1:46070)
0928 02:52:42.520879 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:42.520909 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:42.521065 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:42.521568 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:42.521572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:42.521581 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:52:42.521820 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:42.521827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.523435 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.523440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:42.524597 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:42.524609 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:42.525685 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:42.525691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:42.525843 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:42.525881 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:52:42.541220 (+ 15339us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 02:52:42.541420 (+ 200us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.229.254:41273: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":375,"threads_started":1}
W20260928 02:52:42.541764 1443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46070: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36963/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 02:52:42.543159 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1374
I20260928 02:52:42.544377 1435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:42.637528 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1374
2026-09-28T02:52:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (746 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 02:52:42.693562 28567 mini_oidc.cc:152] Starting JWKS server
I20260928 02:52:42.695785 28567 webserver.cc:460] Webserver started at https://127.0.0.1:44131/ using document root <none> and password file <none>
I20260928 02:52:42.696076 28567 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 02:52:42.696795 28567 webserver.cc:460] Webserver started at http://127.0.0.1:36967/ using document root <none> and password file <none>
2026-09-28T02:52:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:42Z Disabled control of system clock
I20260928 02:52:42.726372 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34625
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:46439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34625
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44131/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:52:43.057538 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:43.057863 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:43.057969 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:43.065382 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:43.065500 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:43.065539 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:43.065588 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:43.065632 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:52:43.065688 1477 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:52:43.065726 1477 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44131/jwks.json
I20260928 02:52:43.075618 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44131/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:43.076932 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:43.078815 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:43.091396 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:43.091383 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:43.092264 1477 server_base.cc:1034] running on GCE node
W20260928 02:52:43.092326 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:43.093305 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:43.094956 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:43.096226 1477 hybrid_clock.cc:648] HybridClock initialized: now 1790563963096181 us; error 41 us; skew 500 ppm
I20260928 02:52:43.096635 1477 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:43.099498 1477 webserver.cc:460] Webserver started at http://127.27.229.254:43193/ using document root <none> and password file <none>
I20260928 02:52:43.100252 1477 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:43.100390 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:43.100735 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:43.103812 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "1ac7a3de13034aa8bc8232af48928382"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "63ea3ddf5c5165aafcac00fa69dc9a00"
server_key_iv: "b5c8f3e3fcab262e6341525604d02532"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.104645 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "1ac7a3de13034aa8bc8232af48928382"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "63ea3ddf5c5165aafcac00fa69dc9a00"
server_key_iv: "b5c8f3e3fcab262e6341525604d02532"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.110056 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:52:43.113708 1491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:43.115180 1477 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:43.115373 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "1ac7a3de13034aa8bc8232af48928382"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "63ea3ddf5c5165aafcac00fa69dc9a00"
server_key_iv: "b5c8f3e3fcab262e6341525604d02532"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.115617 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:43.135389 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:43.136319 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:43.136711 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:43.156745 1477 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34625
I20260928 02:52:43.156780 1542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34625 every 8 connection(s)
I20260928 02:52:43.158229 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:43.162099 1543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:43.167600 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1477
I20260928 02:52:43.167814 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:43.168553 28567 external_mini_cluster.cc:1428] Setting key 49c017f5767b4f80d6862ad043f6b02a
I20260928 02:52:43.174088 1543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382: Bootstrap starting.
I20260928 02:52:43.177333 1543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:43.178363 1543 log.cc:824] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:43.181702 1543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382: No bootstrap required, opened a new log
I20260928 02:52:43.182605 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:43.170882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55846 (local address 127.27.229.254:34625)
0928 02:52:43.172061 (+ 1179us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:43.172074 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:43.172120 (+ 46us) server_negotiation.cc:407] Connection header received
0928 02:52:43.173767 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:43.173790 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:43.174310 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:43.174754 (+ 444us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:43.175338 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.176640 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.178022 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.178775 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.179024 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:43.180235 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:43.180242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:43.180258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:43.180802 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:43.180881 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:43.180918 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:43.181010 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:43.181256 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:52:43.181586 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":392,"threads_started":1}
I20260928 02:52:43.187009 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:52:43.190683 1543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } }
I20260928 02:52:43.191121 1543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:43.191241 1543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ac7a3de13034aa8bc8232af48928382, State: Initialized, Role: FOLLOWER
I20260928 02:52:43.191811 1543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } }
I20260928 02:52:43.192035 1543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:43.192188 1543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:43.192355 1543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:43.194983 1543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } }
I20260928 02:52:43.195499 1543 leader_election.cc:302] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ac7a3de13034aa8bc8232af48928382; no voters:
I20260928 02:52:43.196148 1543 leader_election.cc:288] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:43.196300 1548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:43.197013 1548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [term 1 LEADER]: Becoming Leader. State: Replica: 1ac7a3de13034aa8bc8232af48928382, State: Running, Role: LEADER
I20260928 02:52:43.197515 1548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } }
I20260928 02:52:43.198264 1543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:43.200706 1550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ac7a3de13034aa8bc8232af48928382. Latest consensus state: current_term: 1 leader_uuid: "1ac7a3de13034aa8bc8232af48928382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } } }
I20260928 02:52:43.201151 1550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:43.201067 1549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ac7a3de13034aa8bc8232af48928382" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ac7a3de13034aa8bc8232af48928382" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34625 } } }
I20260928 02:52:43.201431 1549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:43.203015 1554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:43.208209 1554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:43.216915 1554 catalog_manager.cc:1347] Generated new cluster ID: 64c8621529e941c2834cac920c25e864
I20260928 02:52:43.217051 1554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:43.240054 1554 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:43.241581 1554 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:43.254010 1554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ac7a3de13034aa8bc8232af48928382: Generated new TSK 0
I20260928 02:52:43.255200 1554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:52:43.281036 1546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:43.272168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.27.229.254:34625)
0928 02:52:43.272371 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:43.272377 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:43.272788 (+ 411us) server_negotiation.cc:407] Connection header received
0928 02:52:43.272998 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:43.273003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:43.273115 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:43.273290 (+ 175us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:52:43.273772 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.274964 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.276463 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.277160 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.277286 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:43.277955 (+ 669us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:52:43.280376 (+ 2421us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:52:43.280711 (+ 335us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":75}
W20260928 02:52:43.281168 1546 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55858: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 02:52:43.283180 1572 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:43.272027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34625 (local address 127.0.0.1:55858)
0928 02:52:43.272681 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:43.272698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:43.272828 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:43.273307 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:43.273311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:43.273321 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:52:43.273621 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:43.273628 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.275110 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.275115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:43.276318 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:43.276330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.277389 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.277397 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:43.277622 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:43.277659 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:52:43.280597 (+ 2938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 02:52:43.282681 (+ 2084us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.229.254:34625: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44131/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":533,"mutex_wait_us":1752,"thread_start_us":359,"threads_started":1}
I20260928 02:52:43.285004 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1477
I20260928 02:52:43.286319 1538 generic_service.cc:196] Checking for leaks (request via RPC)
E20260928 02:52:43.294804 1573 webserver.cc:533] Webserver: sslize failed:
E20260928 02:52:43.294992 1573 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 02:52:43.379031 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1477
2026-09-28T02:52:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (742 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T02:52:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:43Z Disabled control of system clock
I20260928 02:52:43.463366 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:43843
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:42315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:43843
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/testchainca.cert with env {}
W20260928 02:52:43.796377 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:43.796710 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:43.796876 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:43.804490 1578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:43.804618 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:43.804690 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:43.804800 1578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:43.804869 1578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 02:52:43.804991 1578 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 02:52:43.805051 1578 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 02:52:43.815472 1578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:42315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:43843
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:43843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:43.816854 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:43.818161 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:43.830431 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:43.830401 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:43.830374 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:43.831326 1578 server_base.cc:1034] running on GCE node
I20260928 02:52:43.832291 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:43.833889 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:43.835139 1578 hybrid_clock.cc:648] HybridClock initialized: now 1790563963835090 us; error 38 us; skew 500 ppm
I20260928 02:52:43.835512 1578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:52:43.838196 1578 webserver.cc:460] Webserver started at http://127.27.229.254:46165/ using document root <none> and password file <none>
I20260928 02:52:43.839015 1578 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:43.839154 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:43.839475 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:43.842485 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "f8231a5b56154088bea9eaf41d8b8427"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "7534f877ddcb4601e5a7a7198dbaddbf"
server_key_iv: "67fcd6e86e8d5d0fb10b92abf9334c95"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.843374 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "f8231a5b56154088bea9eaf41d8b8427"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "7534f877ddcb4601e5a7a7198dbaddbf"
server_key_iv: "67fcd6e86e8d5d0fb10b92abf9334c95"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.848730 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:43.852427 1592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:43.853933 1578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:52:43.854168 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "f8231a5b56154088bea9eaf41d8b8427"
format_stamp: "Formatted at 2026-09-28 02:52:43 on dist-test-slave-s1kg"
server_key: "7534f877ddcb4601e5a7a7198dbaddbf"
server_key_iv: "67fcd6e86e8d5d0fb10b92abf9334c95"
server_key_version: "encryptionkey@0"
I20260928 02:52:43.854413 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:43.896705 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:43.897608 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:43.897920 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:43.917382 1578 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:43843
I20260928 02:52:43.917387 1643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:43843 every 8 connection(s)
I20260928 02:52:43.918926 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:43.922943 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:43.925364 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1578
I20260928 02:52:43.925603 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:43.926239 28567 external_mini_cluster.cc:1428] Setting key 5f1ed25df7e16c2bcf8d8d33a790f795
I20260928 02:52:43.935122 1644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427: Bootstrap starting.
I20260928 02:52:43.938753 1644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:43.941715 1644 log.cc:824] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:43.942687 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:43.928361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52762 (local address 127.27.229.254:43843)
0928 02:52:43.929604 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:43.929616 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:43.929661 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:52:43.931607 (+ 1946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:43.931631 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:43.932202 (+ 571us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:43.932684 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:43.933357 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.934744 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.936549 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:43.937430 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:43.937647 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:43.939137 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:43.939146 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:43.939191 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:52:43.939986 (+ 795us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:43.940095 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:52:43.940119 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:43.940258 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:43.940846 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 02:52:43.941219 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":391,"threads_started":1}
I20260928 02:52:43.945298 1644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427: No bootstrap required, opened a new log
I20260928 02:52:43.947556 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:52:43.955924 1644 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } }
I20260928 02:52:43.956362 1644 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:43.956447 1644 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8231a5b56154088bea9eaf41d8b8427, State: Initialized, Role: FOLLOWER
I20260928 02:52:43.957026 1644 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } }
I20260928 02:52:43.957247 1644 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:43.957368 1644 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:43.957521 1644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:43.960150 1644 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } }
I20260928 02:52:43.960721 1644 leader_election.cc:302] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8231a5b56154088bea9eaf41d8b8427; no voters:
I20260928 02:52:43.961349 1644 leader_election.cc:288] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:43.961593 1649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:43.963091 1649 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [term 1 LEADER]: Becoming Leader. State: Replica: f8231a5b56154088bea9eaf41d8b8427, State: Running, Role: LEADER
I20260928 02:52:43.963621 1649 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } }
I20260928 02:52:43.963904 1644 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:43.966701 1651 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8231a5b56154088bea9eaf41d8b8427. Latest consensus state: current_term: 1 leader_uuid: "f8231a5b56154088bea9eaf41d8b8427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } } }
I20260928 02:52:43.966742 1650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8231a5b56154088bea9eaf41d8b8427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8231a5b56154088bea9eaf41d8b8427" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 43843 } } }
I20260928 02:52:43.967180 1650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:43.967180 1651 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:43.968003 1656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:43.973510 1656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:43.983469 1656 catalog_manager.cc:1347] Generated new cluster ID: cd193634283a40e0b3cbed276b0b266d
I20260928 02:52:43.983603 1656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:44.010206 1656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:44.011865 1656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:44.023445 1656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f8231a5b56154088bea9eaf41d8b8427: Generated new TSK 0
I20260928 02:52:44.024547 1656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:52:44.112285 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:44.046597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52766 (local address 127.27.229.254:43843)
0928 02:52:44.046820 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:44.046858 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:44.047332 (+ 474us) server_negotiation.cc:407] Connection header received
0928 02:52:44.047577 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:44.047584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:44.047686 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:44.047884 (+ 198us) server_negotiation.cc:226] Negotiated authn=JWT
0928 02:52:44.048443 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:44.049602 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:44.051193 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:44.051906 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:44.052048 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:44.052549 (+ 501us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 02:52:44.111602 (+ 59053us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:52:44.112006 (+ 404us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:52:44.112457 1647 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52766: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 02:52:44.112707 1673 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:44.046413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:43843 (local address 127.0.0.1:52766)
0928 02:52:44.047212 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:44.047231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:44.047380 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:44.047965 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:44.047969 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:44.047980 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0928 02:52:44.048281 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:44.048289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:44.049775 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:44.049779 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:44.051038 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:44.051052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:44.052120 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:44.052125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:44.052247 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:44.052282 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 02:52:44.111945 (+ 59663us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 02:52:44.112265 (+ 320us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.229.254:43843: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":465,"threads_started":1}
I20260928 02:52:44.114578 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1578
I20260928 02:52:44.116081 1639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:44.215732 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1578
2026-09-28T02:52:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (815 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (416 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (421 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:45 dist-test-slave-s1kg krb5kdc[1688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:45 dist-test-slave-s1kg krb5kdc[1688](info): set up 2 sockets
Sep 28 02:52:45 dist-test-slave-s1kg krb5kdc[1688](info): commencing operation
krb5kdc: starting...
W20260928 02:52:46.184621 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:46 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:52:46.418433 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37737
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:37737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:46.756351 1701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:46.756685 1701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:46.756841 1701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:46.764384 1701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:46.764518 1701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:46.764591 1701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:46.764678 1701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:46.764763 1701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:46.775101 1701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:37737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:46.776502 1701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:46.777832 1701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:46.790115 1706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:46.790030 1707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:46.790055 1709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:46.791160 1701 server_base.cc:1034] running on GCE node
I20260928 02:52:46.792043 1701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:46.793614 1701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:46.794902 1701 hybrid_clock.cc:648] HybridClock initialized: now 1790563966794862 us; error 62 us; skew 500 ppm
I20260928 02:52:46.795269 1701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:46 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563966, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:46.798506 1701 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:52:46.800132 1701 webserver.cc:460] Webserver started at http://127.27.229.254:45293/ using document root <none> and password file <none>
I20260928 02:52:46.800840 1701 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:46.800983 1701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:46.801306 1701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:46.804394 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "50976f4e575a4f4db930044ccf9fa4ca"
format_stamp: "Formatted at 2026-09-28 02:52:46 on dist-test-slave-s1kg"
server_key: "d0fd934768a3e9031d0e0645c5310080"
server_key_iv: "c18fde4eb0405ebd9909ece215f69faa"
server_key_version: "encryptionkey@0"
I20260928 02:52:46.805191 1701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "50976f4e575a4f4db930044ccf9fa4ca"
format_stamp: "Formatted at 2026-09-28 02:52:46 on dist-test-slave-s1kg"
server_key: "d0fd934768a3e9031d0e0645c5310080"
server_key_iv: "c18fde4eb0405ebd9909ece215f69faa"
server_key_version: "encryptionkey@0"
I20260928 02:52:46.810549 1701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:52:46.814097 1716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:46.815693 1701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:46.815894 1701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "50976f4e575a4f4db930044ccf9fa4ca"
format_stamp: "Formatted at 2026-09-28 02:52:46 on dist-test-slave-s1kg"
server_key: "d0fd934768a3e9031d0e0645c5310080"
server_key_iv: "c18fde4eb0405ebd9909ece215f69faa"
server_key_version: "encryptionkey@0"
I20260928 02:52:46.816120 1701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:46.845687 1701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:46.850807 1701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:46.851177 1701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:46.871008 1701 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:37737
I20260928 02:52:46.871029 1767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:37737 every 8 connection(s)
I20260928 02:52:46.872440 1701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:46.876384 1768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:46.880844 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1701
I20260928 02:52:46.881064 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:46.881760 28567 external_mini_cluster.cc:1428] Setting key fad7b96d4289c32937242c6fef1b2aaa
I20260928 02:52:46.890653 1768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca: Bootstrap starting.
I20260928 02:52:46.894953 1768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:46.896495 1768 log.cc:824] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:46.900345 1768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca: No bootstrap required, opened a new log
Sep 28 02:52:46 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:46.911867 1771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:46.884124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47706 (local address 127.27.229.254:37737)
0928 02:52:46.885018 (+ 894us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:46.885038 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:46.885107 (+ 69us) server_negotiation.cc:407] Connection header received
0928 02:52:46.888586 (+ 3479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:46.888627 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:46.889259 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:46.889794 (+ 535us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:46.891300 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:46.893587 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:46.895095 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:46.896331 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:46.896561 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:46.901496 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:46.901508 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:46.901530 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:46.901594 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:46.906280 (+ 4686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:46.907247 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:46.907258 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:46.907268 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:46.907535 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:46.907972 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:46.907982 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:46.907986 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:46.908899 (+ 913us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:46.909311 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:46.909639 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:52:46.910200 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":368,"threads_started":1}
I20260928 02:52:46.912634 1768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } }
I20260928 02:52:46.913064 1768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:46.913190 1768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50976f4e575a4f4db930044ccf9fa4ca, State: Initialized, Role: FOLLOWER
I20260928 02:52:46.913961 1768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } }
I20260928 02:52:46.914280 1768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:46.914435 1768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:46.914635 1768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:46.917161 1768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } }
I20260928 02:52:46.917691 1768 leader_election.cc:302] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50976f4e575a4f4db930044ccf9fa4ca; no voters:
I20260928 02:52:46.918427 1768 leader_election.cc:288] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:46.918558 1773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:46.919627 1773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [term 1 LEADER]: Becoming Leader. State: Replica: 50976f4e575a4f4db930044ccf9fa4ca, State: Running, Role: LEADER
I20260928 02:52:46.920418 1773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } }
I20260928 02:52:46.921128 1768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:46.923641 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50976f4e575a4f4db930044ccf9fa4ca. Latest consensus state: current_term: 1 leader_uuid: "50976f4e575a4f4db930044ccf9fa4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } } }
I20260928 02:52:46.923858 1774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50976f4e575a4f4db930044ccf9fa4ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50976f4e575a4f4db930044ccf9fa4ca" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37737 } } }
I20260928 02:52:46.924132 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:46.924196 1774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:46.925125 1780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:46.932189 1780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:46.941874 1780 catalog_manager.cc:1347] Generated new cluster ID: b5c97e0b2aba4a34a1b246ea84b49e22
I20260928 02:52:46.942027 1780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:46.953755 1780 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:46.955794 1780 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:46.965440 1780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50976f4e575a4f4db930044ccf9fa4ca: Generated new TSK 0
I20260928 02:52:46.966675 1780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:52:47.034629 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:47.388672 1794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:47.389014 1794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:47.389173 1794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:47.396777 1794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:47.396944 1794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:47.397117 1794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:47.408954 1794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:47.410351 1794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:47.411752 1794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:47.426071 1800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:47.426023 1799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:47.426608 1802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:47.426944 1794 server_base.cc:1034] running on GCE node
I20260928 02:52:47.427554 1794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:47.428581 1794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:47.429793 1794 hybrid_clock.cc:648] HybridClock initialized: now 1790563967429706 us; error 96 us; skew 500 ppm
I20260928 02:52:47.430100 1794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:47 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:47.436331 1794 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:52:47.437815 1794 webserver.cc:460] Webserver started at http://127.27.229.193:37071/ using document root <none> and password file <none>
I20260928 02:52:47.438536 1794 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:47.438637 1794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:47.438963 1794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:47.441920 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "96c7f3a5769c40d0a0856c2476d638fe"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "79a47045bdc0da86ef0960683a59f447"
server_key_iv: "352c864c49eb92f0dfc342984db26dbf"
server_key_version: "encryptionkey@0"
I20260928 02:52:47.442714 1794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "96c7f3a5769c40d0a0856c2476d638fe"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "79a47045bdc0da86ef0960683a59f447"
server_key_iv: "352c864c49eb92f0dfc342984db26dbf"
server_key_version: "encryptionkey@0"
I20260928 02:52:47.448392 1794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:52:47.452056 1809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:47.453560 1794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:52:47.453711 1794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "96c7f3a5769c40d0a0856c2476d638fe"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "79a47045bdc0da86ef0960683a59f447"
server_key_iv: "352c864c49eb92f0dfc342984db26dbf"
server_key_version: "encryptionkey@0"
I20260928 02:52:47.453889 1794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:47.481292 1794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:47.486759 1794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:47.487224 1794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:47.488307 1794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:47.489953 1794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:47.490041 1794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:47.490139 1794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:47.490187 1794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:47.526796 1794 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:38091
I20260928 02:52:47.526849 1921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:38091 every 8 connection(s)
I20260928 02:52:47.528148 1794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:47.530107 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1794
I20260928 02:52:47.530300 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:47.531098 28567 external_mini_cluster.cc:1428] Setting key 538e5a6f97eaf0acc5234a421073de6d
Sep 28 02:52:47 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:47.557766 1924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:47.531053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:49131 (local address 127.27.229.254:37737)
0928 02:52:47.539280 (+ 8227us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:47.539284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:47.539303 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:52:47.539356 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:47.539360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:47.539431 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:47.539524 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:47.541897 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:47.542740 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:47.545569 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:47.546030 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:47.546157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:47.551119 (+ 4962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:47.551126 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:47.551130 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:47.551169 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:47.554043 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:47.554982 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:47.554987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:47.554989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:47.555111 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:47.555664 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:47.555670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:47.555673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:47.555948 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:47.556060 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:47.557220 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0928 02:52:47.557450 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8109,"thread_start_us":7984,"threads_started":1}
I20260928 02:52:47.559504 1925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:47.531589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37737 (local address 127.27.229.193:49131)
0928 02:52:47.532641 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:47.532699 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:47.536075 (+ 3376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:47.539882 (+ 3807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:47.539898 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:47.540772 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:47.541657 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:47.541687 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:47.542990 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:47.542998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:47.545351 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:47.545371 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:47.546292 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:47.546301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:47.546665 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:47.547520 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:47.547555 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:47.550767 (+ 3212us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:47.554220 (+ 3453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:47.554228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:47.554244 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:47.554761 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:47.555258 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:47.555261 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:47.555264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:47.555528 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:47.556107 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:47.556119 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:47.556664 (+ 545us) client_negotiation.cc:769] Sending connection context
0928 02:52:47.557016 (+ 352us) client_negotiation.cc:240] Negotiation successful
0928 02:52:47.557786 (+ 770us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":410,"threads_started":1}
I20260928 02:52:47.561241 1922 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37737
I20260928 02:52:47.561583 1922 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:47.562342 1922 heartbeater.cc:511] Master 127.27.229.254:37737 requested a full tablet report, sending...
I20260928 02:52:47.564718 1733 ts_manager.cc:194] Registered new tserver with Master: 96c7f3a5769c40d0a0856c2476d638fe (127.27.229.193:38091)
I20260928 02:52:47.566293 1733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:49131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:52:47.586239 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:47.942792 1928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:47.943202 1928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:47.943377 1928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:47.950702 1928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:47.950918 1928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:47.951128 1928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:47.961313 1928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:47.962592 1928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:47.963948 1928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:47.977550 1933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:47.977641 1934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:47.977638 1936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:47.978329 1928 server_base.cc:1034] running on GCE node
I20260928 02:52:47.978991 1928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:47.979943 1928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:47.981199 1928 hybrid_clock.cc:648] HybridClock initialized: now 1790563967981143 us; error 46 us; skew 500 ppm
I20260928 02:52:47.981562 1928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:47 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:47.984802 1928 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:52:47.986392 1928 webserver.cc:460] Webserver started at http://127.27.229.194:42333/ using document root <none> and password file <none>
I20260928 02:52:47.987177 1928 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:47.987282 1928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:47.987555 1928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:47.990447 1928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "56e4b38ec32c47c6a797b4f272b4f059"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "3e4c86b68db76c47527b3898adc90aca"
server_key_iv: "396669ad820ad2eff724a0ad4f384377"
server_key_version: "encryptionkey@0"
I20260928 02:52:47.991269 1928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "56e4b38ec32c47c6a797b4f272b4f059"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "3e4c86b68db76c47527b3898adc90aca"
server_key_iv: "396669ad820ad2eff724a0ad4f384377"
server_key_version: "encryptionkey@0"
I20260928 02:52:47.996387 1928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:52:48.000203 1943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.001896 1928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:48.002111 1928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "56e4b38ec32c47c6a797b4f272b4f059"
format_stamp: "Formatted at 2026-09-28 02:52:47 on dist-test-slave-s1kg"
server_key: "3e4c86b68db76c47527b3898adc90aca"
server_key_iv: "396669ad820ad2eff724a0ad4f384377"
server_key_version: "encryptionkey@0"
I20260928 02:52:48.002337 1928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:48.023452 1928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:48.028755 1928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:48.029084 1928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:48.030130 1928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:48.031499 1928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:48.031603 1928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.031746 1928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:48.031816 1928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.066455 1928 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:40437
I20260928 02:52:48.066495 2055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:40437 every 8 connection(s)
I20260928 02:52:48.067885 1928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:48.068876 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 1928
I20260928 02:52:48.069052 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:48.069620 28567 external_mini_cluster.cc:1428] Setting key 1466ac9ca79d466d785112b287e320e0
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563967, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:52:48.094676 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.070958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:52393 (local address 127.27.229.254:37737)
0928 02:52:48.071447 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.071453 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.072793 (+ 1340us) server_negotiation.cc:407] Connection header received
0928 02:52:48.075972 (+ 3179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.075976 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.076053 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.076155 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.078518 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.079691 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.081439 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.082338 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.082502 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.086565 (+ 4063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.086576 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.086585 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.086648 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.090990 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.091957 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.091964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.091968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.092151 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.092658 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.092664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.092668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.093093 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.093275 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.094002 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.094278 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260928 02:52:48.095652 2059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.071623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37737 (local address 127.27.229.194:52393)
0928 02:52:48.072613 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:48.072668 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:48.075495 (+ 2827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:48.076478 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:48.076494 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:48.077377 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:48.078256 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:48.078286 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.079928 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.079932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:48.081269 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:48.081283 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.082574 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.082580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:48.082805 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.083621 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:48.083653 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:48.086263 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:48.091258 (+ 4995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:48.091266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:48.091282 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:48.091808 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:48.092294 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:48.092298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:48.092300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:48.092498 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:48.093244 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:48.093251 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:48.093615 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 02:52:48.093867 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 02:52:48.094370 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":410,"threads_started":1}
I20260928 02:52:48.097317 2056 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:52:48.097664 2056 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:48.098673 2056 heartbeater.cc:511] Master 127.27.229.254:37737 requested a full tablet report, sending...
I20260928 02:52:48.100461 1733 ts_manager.cc:194] Registered new tserver with Master: 56e4b38ec32c47c6a797b4f272b4f059 (127.27.229.194:40437)
I20260928 02:52:48.101832 1733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:52393
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:52:48.127416 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:48.471119 2062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:48.471470 2062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:48.471642 2062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:48.479166 2062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:48.479321 2062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:48.479494 2062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:48.489454 2062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:48.490798 2062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:48.492146 2062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:48.505364 2070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:48.505426 2068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:48.505450 2067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:48.506297 2062 server_base.cc:1034] running on GCE node
I20260928 02:52:48.507054 2062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:48.508028 2062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:48.509259 2062 hybrid_clock.cc:648] HybridClock initialized: now 1790563968509210 us; error 36 us; skew 500 ppm
I20260928 02:52:48.509637 2062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.513051 2062 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:52:48.514587 2062 webserver.cc:460] Webserver started at http://127.27.229.195:39945/ using document root <none> and password file <none>
I20260928 02:52:48.515332 2062 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:48.515434 2062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:48.515702 2062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:48.518560 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "1c1430f5eae64a71b0698010ac7a966a"
format_stamp: "Formatted at 2026-09-28 02:52:48 on dist-test-slave-s1kg"
server_key: "22009ea1a701c27ce9158f525a2058fc"
server_key_iv: "0e3353503a0ff2e13fd89a1948c83989"
server_key_version: "encryptionkey@0"
I20260928 02:52:48.519393 2062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "1c1430f5eae64a71b0698010ac7a966a"
format_stamp: "Formatted at 2026-09-28 02:52:48 on dist-test-slave-s1kg"
server_key: "22009ea1a701c27ce9158f525a2058fc"
server_key_iv: "0e3353503a0ff2e13fd89a1948c83989"
server_key_version: "encryptionkey@0"
I20260928 02:52:48.524922 2062 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:52:48.528740 2077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.530174 2062 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:48.530335 2062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "1c1430f5eae64a71b0698010ac7a966a"
format_stamp: "Formatted at 2026-09-28 02:52:48 on dist-test-slave-s1kg"
server_key: "22009ea1a701c27ce9158f525a2058fc"
server_key_iv: "0e3353503a0ff2e13fd89a1948c83989"
server_key_version: "encryptionkey@0"
I20260928 02:52:48.530539 2062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:48.552758 2062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:48.557745 2062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:48.557999 2062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:48.559207 2062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:48.560928 2062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:48.561029 2062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.561187 2062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:48.561244 2062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:48.570294 1922 heartbeater.cc:503] Master 127.27.229.254:37737 was elected leader, sending a full tablet report...
I20260928 02:52:48.598464 2062 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:38373
I20260928 02:52:48.598554 2189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:38373 every 8 connection(s)
I20260928 02:52:48.599912 2062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:48.601356 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2062
I20260928 02:52:48.601563 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:48.602100 28567 external_mini_cluster.cc:1428] Setting key 082ab48b8d2be856c33fa578700a72d6
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:52:48.628090 2192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.603038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:43447 (local address 127.27.229.254:37737)
0928 02:52:48.611247 (+ 8209us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.611252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.611273 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:48.611326 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.611330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.611394 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.611532 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.613312 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.614101 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.615770 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.616179 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.616307 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.621106 (+ 4799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.621111 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.621115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.621163 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.624156 (+ 2993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.625079 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.625083 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.625086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.625228 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.625691 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.625695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.625697 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.626124 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.626291 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.627394 (+ 1103us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.627679 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8094,"thread_start_us":7985,"threads_started":1}
I20260928 02:52:48.628707 2193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.603693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37737 (local address 127.27.229.195:43447)
0928 02:52:48.604746 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:48.604797 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:48.607078 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:48.611855 (+ 4777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:48.611867 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:48.612456 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:48.613110 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:48.613130 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.614255 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.614260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:48.615602 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:48.615615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.616497 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.616505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:48.616892 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.617950 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:48.617980 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:48.620850 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:48.624344 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:48.624353 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:48.624369 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:48.624909 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:48.625328 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:48.625332 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:48.625334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:48.625553 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:48.626288 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:48.626295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:48.626647 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 02:52:48.627068 (+ 421us) client_negotiation.cc:240] Negotiation successful
0928 02:52:48.627528 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":420,"threads_started":1}
I20260928 02:52:48.630390 2190 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37737
I20260928 02:52:48.631156 2190 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:48.632053 2190 heartbeater.cc:511] Master 127.27.229.254:37737 requested a full tablet report, sending...
I20260928 02:52:48.633929 1732 ts_manager.cc:194] Registered new tserver with Master: 1c1430f5eae64a71b0698010ac7a966a (127.27.229.195:38373)
I20260928 02:52:48.634994 1732 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:43447
I20260928 02:52:48.645260 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/test-admin.keytab.
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.673838 28567 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 02:52:48.679221 2198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:48.676660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55338 (local address 127.27.229.193:38091)
0928 02:52:48.677327 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.677334 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.677360 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:52:48.677420 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.677426 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.677647 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.677791 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.678725 (+ 934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55338: BlockingRecv error: recv got EOF from 127.0.0.1:55338 (error 108)
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":393,"threads_started":1}
W20260928 02:52:48.679447 28567 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.681157 28567 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.682544 28567 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:48.699306 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.684546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.27.229.193:38091)
0928 02:52:48.684744 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.684749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.684874 (+ 125us) server_negotiation.cc:407] Connection header received
0928 02:52:48.685096 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.685100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.685163 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.685250 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.686359 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.687407 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.688681 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.689270 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.689378 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.693684 (+ 4306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.693707 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.693723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.693770 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.696685 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.697395 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.697402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.697405 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.697543 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.697859 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.697863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.697865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.698220 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.698429 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.698684 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.699013 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:52:48.705363 2198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:48.703541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.27.229.193:38091)
0928 02:52:48.703755 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.703760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.703945 (+ 185us) server_negotiation.cc:407] Connection header received
0928 02:52:48.704172 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.704176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.704233 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.704331 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.705111 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55364: BlockingRecv error: recv got EOF from 127.0.0.1:55364 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 02:52:48.705552 28567 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.707481 28567 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.708994 28567 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:48.724952 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.710406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55368 (local address 127.27.229.193:38091)
0928 02:52:48.710659 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.710666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.710998 (+ 332us) server_negotiation.cc:407] Connection header received
0928 02:52:48.711202 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.711207 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.711281 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.711383 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.712324 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.713090 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.714311 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.714743 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.714934 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.718645 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.718650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.718654 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.718699 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.722131 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.722932 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.722938 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.722942 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.723094 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.723531 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.723536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.723540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.723832 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.724117 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.724432 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.724648 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 02:52:48.729481 2198 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:52:48.727807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55376 (local address 127.27.229.193:38091)
0928 02:52:48.728030 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.728035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.728249 (+ 214us) server_negotiation.cc:407] Connection header received
0928 02:52:48.728378 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.728381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.728460 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.728540 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.729224 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55376: BlockingRecv error: recv got EOF from 127.0.0.1:55376 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 02:52:48.729555 28567 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790563915649057-28567-0/krb5kdc/krb5cc' of type FILE
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.731127 28567 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:48.732364 28567 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:52:48.747937 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.733717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.27.229.193:38091)
0928 02:52:48.733951 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.733958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.734283 (+ 325us) server_negotiation.cc:407] Connection header received
0928 02:52:48.734449 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.734453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.734516 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.734607 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.735757 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.736503 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.737783 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.738213 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.738308 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.741787 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.741792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.741796 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.741833 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.744769 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.745509 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.745514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.745516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.745646 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.746037 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.746041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.746043 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.746328 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.746577 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.747250 (+ 673us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.747442 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:52:48.749771 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1794
I20260928 02:52:48.763384 2198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.750617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55404 (local address 127.27.229.193:38091)
0928 02:52:48.750776 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.750781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.751093 (+ 312us) server_negotiation.cc:407] Connection header received
0928 02:52:48.751209 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.751213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.751276 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.751370 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.752393 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.753126 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.754376 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.754884 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.755000 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.757355 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.757360 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.757363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.757398 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.760438 (+ 3040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.761394 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.761399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.761402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.761555 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.761886 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.761889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.761891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.762188 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.762377 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.762720 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.762986 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:52:48.764019 1917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:48.849321 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1794
I20260928 02:52:48.865939 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1928
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:52:48.885197 2211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.866807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43794 (local address 127.27.229.194:40437)
0928 02:52:48.867519 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.867527 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.867552 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:52:48.867646 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.867656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.867964 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.868147 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.869468 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.871577 (+ 2109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.873862 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.874539 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.874706 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.879313 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.879328 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.879344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.879393 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:48.882192 (+ 2799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.883045 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.883054 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.883058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.883210 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:48.883582 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:48.883588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:48.883591 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:48.883946 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:48.884285 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:48.884599 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:52:48.884880 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":341,"threads_started":1}
I20260928 02:52:48.886504 2051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:48.972361 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1928
I20260928 02:52:48.988523 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2062
Sep 28 02:52:48 dist-test-slave-s1kg krb5kdc[1688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563968, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:52:49.004834 2213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:48.989498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.27.229.195:38373)
0928 02:52:48.990086 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:48.990092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:48.990121 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:52:48.990178 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:48.990185 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:48.990399 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:48.990599 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:48.991721 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.992889 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.994156 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:48.994758 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:48.994967 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:48.998951 (+ 3984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:48.998964 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:48.998979 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:48.999022 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:49.001991 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:49.002705 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:49.002712 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:49.002715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:49.002924 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:49.003298 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:49.003301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:49.003303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:49.003672 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:49.003920 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:49.004209 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 02:52:49.004504 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":294,"threads_started":1}
I20260928 02:52:49.006538 2185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:49.090372 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2062
I20260928 02:52:49.106252 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 1701
I20260928 02:52:49.107628 1763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:49.210381 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 1701
2026-09-28T02:52:49Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4161 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:49 dist-test-slave-s1kg krb5kdc[2218](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:49 dist-test-slave-s1kg krb5kdc[2218](info): set up 2 sockets
Sep 28 02:52:49 dist-test-slave-s1kg krb5kdc[2218](info): commencing operation
krb5kdc: starting...
W20260928 02:52:50.327843 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:52:50 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:52:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:52:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab.
Entry for principal oryx/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab.
I20260928 02:52:50.558640 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42995
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab
--principal=oryx/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:42995
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:50.893026 2231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:50.893361 2231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:50.893523 2231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:50.901106 2231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:50.901252 2231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:50.901322 2231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:50.901392 2231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:50.901456 2231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:50.911844 2231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:42995
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab
--principal=oryx/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:50.913196 2231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:50.914541 2231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:50.926688 2239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:50.926738 2237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:50.926774 2236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:50.927623 2231 server_base.cc:1034] running on GCE node
I20260928 02:52:50.928473 2231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:50.929940 2231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:50.931173 2231 hybrid_clock.cc:648] HybridClock initialized: now 1790563970931130 us; error 39 us; skew 500 ppm
I20260928 02:52:50.931551 2231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:50 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:50.934741 2231 init.cc:377] Logged in from keytab as oryx/127.27.229.254@KRBTEST.COM (short username oryx)
I20260928 02:52:50.936383 2231 webserver.cc:460] Webserver started at http://127.27.229.254:44377/ using document root <none> and password file <none>
I20260928 02:52:50.937083 2231 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:50.937224 2231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:50.937532 2231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:50.940872 2231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "5a349fb854e649fcb24b3f62de8a1cc3"
format_stamp: "Formatted at 2026-09-28 02:52:50 on dist-test-slave-s1kg"
server_key: "1b1412e15e22176b03fa3d5ad84b2239"
server_key_iv: "ed9dd1679402dad13df3082a32db4128"
server_key_version: "encryptionkey@0"
I20260928 02:52:50.941642 2231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "5a349fb854e649fcb24b3f62de8a1cc3"
format_stamp: "Formatted at 2026-09-28 02:52:50 on dist-test-slave-s1kg"
server_key: "1b1412e15e22176b03fa3d5ad84b2239"
server_key_iv: "ed9dd1679402dad13df3082a32db4128"
server_key_version: "encryptionkey@0"
I20260928 02:52:50.946949 2231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:52:50.950457 2246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:50.951972 2231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:52:50.952142 2231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "5a349fb854e649fcb24b3f62de8a1cc3"
format_stamp: "Formatted at 2026-09-28 02:52:50 on dist-test-slave-s1kg"
server_key: "1b1412e15e22176b03fa3d5ad84b2239"
server_key_iv: "ed9dd1679402dad13df3082a32db4128"
server_key_version: "encryptionkey@0"
I20260928 02:52:50.952337 2231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:50.967468 2231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:50.972447 2231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:50.972807 2231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:50.993229 2231 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:42995
I20260928 02:52:50.993259 2297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:42995 every 8 connection(s)
I20260928 02:52:50.994725 2231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:52:50.998453 2298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:51.000839 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2231
I20260928 02:52:51.001048 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:52:51.001856 28567 external_mini_cluster.cc:1428] Setting key 313e38cb74083d4129d01770f2610813
I20260928 02:52:51.011564 2298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Bootstrap starting.
I20260928 02:52:51.015663 2298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:51.017000 2298 log.cc:824] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:52:51 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:51.021111 2298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: No bootstrap required, opened a new log
I20260928 02:52:51.028285 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:51.004058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45404 (local address 127.27.229.254:42995)
0928 02:52:51.004921 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:51.004934 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:51.004975 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:52:51.006682 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:51.006706 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:51.007191 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:51.007544 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:51.009365 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.010752 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.012543 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.013239 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.013373 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:51.018698 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:51.018710 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:51.018734 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:51.018780 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:51.023130 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:51.023990 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:51.024001 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:51.024017 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:51.024245 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:51.024682 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:51.024688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:51.024691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:51.025478 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:51.025798 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:51.026182 (+ 384us) server_negotiation.cc:299] Negotiation successful
0928 02:52:51.026677 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":403,"threads_started":1}
I20260928 02:52:51.031384 2298 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } }
I20260928 02:52:51.031729 2298 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:51.031810 2298 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a349fb854e649fcb24b3f62de8a1cc3, State: Initialized, Role: FOLLOWER
I20260928 02:52:51.032294 2298 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } }
I20260928 02:52:51.032486 2298 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:52:51.032591 2298 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:52:51.032712 2298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:51.034897 2298 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } }
I20260928 02:52:51.035364 2298 leader_election.cc:302] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a349fb854e649fcb24b3f62de8a1cc3; no voters:
I20260928 02:52:51.036022 2298 leader_election.cc:288] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:52:51.036132 2303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:51.037376 2303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 LEADER]: Becoming Leader. State: Replica: 5a349fb854e649fcb24b3f62de8a1cc3, State: Running, Role: LEADER
I20260928 02:52:51.037904 2303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } }
I20260928 02:52:51.038242 2298 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:51.040930 2305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a349fb854e649fcb24b3f62de8a1cc3. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } }
I20260928 02:52:51.040966 2304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } }
I20260928 02:52:51.041317 2305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:51.041379 2304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:51.042387 2311 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:51.048610 2311 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:51.057986 2311 catalog_manager.cc:1347] Generated new cluster ID: b5f087ef201f423fbebdaba847989d27
I20260928 02:52:51.058146 2311 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:51.068197 2311 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:52:51.069720 2311 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:51.076104 2311 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Generated new TSK 0
I20260928 02:52:51.077210 2311 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
I20260928 02:52:51.139484 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:51.486971 2324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:51.487300 2324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:51.487506 2324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:51.495806 2324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:51.495931 2324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:51.496125 2324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:51.507772 2324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:51.509181 2324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:51.510596 2324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:51.524680 2329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:51.524708 2330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:51.524662 2332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:51.525586 2324 server_base.cc:1034] running on GCE node
I20260928 02:52:51.526247 2324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:51.527356 2324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:51.528630 2324 hybrid_clock.cc:648] HybridClock initialized: now 1790563971528566 us; error 60 us; skew 500 ppm
I20260928 02:52:51.529110 2324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:51 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563971, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:51.533391 2324 init.cc:377] Logged in from keytab as oryx/127.27.229.193@KRBTEST.COM (short username oryx)
I20260928 02:52:51.535341 2324 webserver.cc:460] Webserver started at http://127.27.229.193:34553/ using document root <none> and password file <none>
I20260928 02:52:51.536098 2324 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:51.536260 2324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:51.536580 2324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:51.539942 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "07c3a929c9984276a84754db1d1450a4"
format_stamp: "Formatted at 2026-09-28 02:52:51 on dist-test-slave-s1kg"
server_key: "f3b6cfe880cfe1fd64f1aefb0ff2bdd5"
server_key_iv: "001b7d366d845459a968cb2cf6ad3334"
server_key_version: "encryptionkey@0"
I20260928 02:52:51.540895 2324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "07c3a929c9984276a84754db1d1450a4"
format_stamp: "Formatted at 2026-09-28 02:52:51 on dist-test-slave-s1kg"
server_key: "f3b6cfe880cfe1fd64f1aefb0ff2bdd5"
server_key_iv: "001b7d366d845459a968cb2cf6ad3334"
server_key_version: "encryptionkey@0"
I20260928 02:52:51.547334 2324 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:52:51.551386 2339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:51.553017 2324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:52:51.553215 2324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "07c3a929c9984276a84754db1d1450a4"
format_stamp: "Formatted at 2026-09-28 02:52:51 on dist-test-slave-s1kg"
server_key: "f3b6cfe880cfe1fd64f1aefb0ff2bdd5"
server_key_iv: "001b7d366d845459a968cb2cf6ad3334"
server_key_version: "encryptionkey@0"
I20260928 02:52:51.553431 2324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:51.572641 2324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:51.577713 2324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:51.578069 2324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:51.579118 2324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:51.580551 2324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:51.580655 2324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:51.580801 2324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:51.580868 2324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:51.617993 2324 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:41133
I20260928 02:52:51.618059 2451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:41133 every 8 connection(s)
I20260928 02:52:51.619519 2324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:51.622594 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2324
I20260928 02:52:51.622808 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:52:51.623660 28567 external_mini_cluster.cc:1428] Setting key d99ce5c2aae5cbd74edb84d125d897ff
Sep 28 02:52:51 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563971, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:51.644292 2454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:51.622282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:60493 (local address 127.27.229.254:42995)
0928 02:52:51.622851 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:51.622857 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:51.624018 (+ 1161us) server_negotiation.cc:407] Connection header received
0928 02:52:51.625880 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:51.625885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:51.625947 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:51.626052 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:51.628301 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.629053 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.631728 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.632169 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.632284 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:51.636279 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:51.636288 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:51.636297 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:51.636346 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:51.640148 (+ 3802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:51.641058 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:51.641067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:51.641073 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:51.641232 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:51.641863 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:51.641871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:51.641877 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:51.642278 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:51.642449 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:51.643507 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0928 02:52:51.643790 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":246,"threads_started":1}
I20260928 02:52:51.644987 2455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:51.622887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:60493)
0928 02:52:51.623832 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:51.623885 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:51.625580 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:51.626342 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:51.626355 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:51.627229 (+ 874us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:51.628094 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:51.628123 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.629224 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.629233 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:51.631528 (+ 2295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:51.631548 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:51.632431 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:51.632441 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:51.632729 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:51.633461 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:51.633490 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:51.635979 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:51.640345 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:51.640353 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:51.640370 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:51.640890 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:51.641394 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:51.641398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:51.641400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:51.641726 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:51.642426 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:51.642441 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:51.643047 (+ 606us) client_negotiation.cc:769] Sending connection context
0928 02:52:51.643341 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 02:52:51.643819 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":292,"threads_started":1}
I20260928 02:52:51.646708 2452 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:51.647086 2452 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:52:51.647965 2452 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.194@KRBTEST.COM" created.
I20260928 02:52:51.650386 2263 ts_manager.cc:194] Registered new tserver with Master: 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133)
I20260928 02:52:51.652108 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:60493
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
I20260928 02:52:51.679896 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:52.024546 2458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:52.024902 2458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:52.025068 2458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:52.033288 2458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:52.033500 2458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:52.033710 2458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:52.045501 2458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:52.046996 2458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:52.048440 2458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:52.062870 2463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:52.063114 2466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:52.062894 2464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:52.063637 2458 server_base.cc:1034] running on GCE node
I20260928 02:52:52.064255 2458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:52.065218 2458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:52.066450 2458 hybrid_clock.cc:648] HybridClock initialized: now 1790563972066388 us; error 45 us; skew 500 ppm
I20260928 02:52:52.066804 2458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:52 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563972, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:52.070348 2458 init.cc:377] Logged in from keytab as oryx/127.27.229.194@KRBTEST.COM (short username oryx)
I20260928 02:52:52.072188 2458 webserver.cc:460] Webserver started at http://127.27.229.194:39679/ using document root <none> and password file <none>
I20260928 02:52:52.072958 2458 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:52.073071 2458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:52.073374 2458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:52.076478 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "2c6cf40e3b054e839524736121a2bf11"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "6882a6ae3de3b6d49adfafaae430e21d"
server_key_iv: "83ec06f8448098e3529239c9d2847d7a"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.077297 2458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "2c6cf40e3b054e839524736121a2bf11"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "6882a6ae3de3b6d49adfafaae430e21d"
server_key_iv: "83ec06f8448098e3529239c9d2847d7a"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.083086 2458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 02:52:52.087421 2473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.089735 2458 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:52:52.090018 2458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "2c6cf40e3b054e839524736121a2bf11"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "6882a6ae3de3b6d49adfafaae430e21d"
server_key_iv: "83ec06f8448098e3529239c9d2847d7a"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.090260 2458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:52.115310 2458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:52.120803 2458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:52.121122 2458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:52.122294 2458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:52.123904 2458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:52.123988 2458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.124125 2458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:52.124176 2458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.160043 2458 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:39513
I20260928 02:52:52.160141 2585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:39513 every 8 connection(s)
I20260928 02:52:52.161538 2458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:52.162585 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2458
I20260928 02:52:52.162786 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:52:52.163385 28567 external_mini_cluster.cc:1428] Setting key 42a88c8417c99cfeb0f58580ce1ac837
Sep 28 02:52:52 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563972, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:52.185190 2588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:52.164270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:46799 (local address 127.27.229.254:42995)
0928 02:52:52.164741 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:52.164748 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:52.165959 (+ 1211us) server_negotiation.cc:407] Connection header received
0928 02:52:52.168041 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:52.168047 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:52.168155 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:52.168280 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:52.169928 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.170719 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.172478 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.173340 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.173498 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:52.177071 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:52.177081 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:52.177090 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:52.177149 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:52.181244 (+ 4095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:52.182271 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:52.182280 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:52.182287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:52.182475 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:52.182978 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:52.182987 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:52.182994 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:52.183456 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:52.183654 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:52.184424 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 02:52:52.184693 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":246,"threads_started":1}
I20260928 02:52:52.185865 2589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:52.164927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:46799)
0928 02:52:52.165797 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:52.165842 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:52.167754 (+ 1912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:52.168537 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:52.168547 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:52.169113 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:52.169744 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:52.169762 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.170911 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.170916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:52.172310 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:52.172323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.173583 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.173588 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:52.173802 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:52.174559 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:52.174583 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:52.176809 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:52.181497 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:52.181536 (+ 39us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:52.181559 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:52.182113 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:52.182600 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:52.182603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:52.182605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:52.182777 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:52.183614 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:52.183621 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:52.184003 (+ 382us) client_negotiation.cc:769] Sending connection context
0928 02:52:52.184258 (+ 255us) client_negotiation.cc:240] Negotiation successful
0928 02:52:52.184673 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":371,"threads_started":1}
WARNING: no policy specified for oryx/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:52:52.187902 2586 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:52.188230 2586 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:52.189056 2586 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.195@KRBTEST.COM" created.
I20260928 02:52:52.191388 2263 ts_manager.cc:194] Registered new tserver with Master: 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:52.192652 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:46799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
I20260928 02:52:52.220054 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:52.552444 2592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:52.552781 2592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:52.552955 2592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:52.560528 2592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:52.560639 2592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:52.560824 2592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:52.570648 2592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42995
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:52.572009 2592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:52.573364 2592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:52.587255 2597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:52.587327 2598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:52.587291 2600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:52.588235 2592 server_base.cc:1034] running on GCE node
I20260928 02:52:52.588832 2592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:52.589766 2592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:52.591006 2592 hybrid_clock.cc:648] HybridClock initialized: now 1790563972590935 us; error 57 us; skew 500 ppm
I20260928 02:52:52.591377 2592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:52 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563972, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:52.605274 2592 init.cc:377] Logged in from keytab as oryx/127.27.229.195@KRBTEST.COM (short username oryx)
I20260928 02:52:52.607214 2592 webserver.cc:460] Webserver started at http://127.27.229.195:40269/ using document root <none> and password file <none>
I20260928 02:52:52.607969 2592 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:52.608119 2592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:52.608431 2592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:52.611506 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "695ebf22b5784e60a9830411707b6531"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "27b22beb5bab9b6e1c6194522f7b5a85"
server_key_iv: "841b4a0e0c8691357d80c15f44ed5ccf"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.612322 2592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "695ebf22b5784e60a9830411707b6531"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "27b22beb5bab9b6e1c6194522f7b5a85"
server_key_iv: "841b4a0e0c8691357d80c15f44ed5ccf"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.617669 2592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260928 02:52:52.622197 2607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.623806 2592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:52.623939 2592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "695ebf22b5784e60a9830411707b6531"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "27b22beb5bab9b6e1c6194522f7b5a85"
server_key_iv: "841b4a0e0c8691357d80c15f44ed5ccf"
server_key_version: "encryptionkey@0"
I20260928 02:52:52.624117 2592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:52.642580 2592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:52.647119 2592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:52.647368 2592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:52.648319 2592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:52.649631 2592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:52.649709 2592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.649793 2592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:52.649837 2592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:52.656528 2452 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
I20260928 02:52:52.682727 2592 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43787
I20260928 02:52:52.682762 2719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43787 every 8 connection(s)
I20260928 02:52:52.684088 2592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:52.691910 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2592
I20260928 02:52:52.692104 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:52:52.692561 28567 external_mini_cluster.cc:1428] Setting key 0d9801c17181b144364bbe78055170af
Sep 28 02:52:52 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563972, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:52.704346 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:52.687025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:35383 (local address 127.27.229.254:42995)
0928 02:52:52.687474 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:52.687478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:52.688456 (+ 978us) server_negotiation.cc:407] Connection header received
0928 02:52:52.690247 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:52.690252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:52.690345 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:52.690461 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:52.692022 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.692898 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.694598 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.695107 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.695273 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:52.698762 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:52.698773 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:52.698776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:52.698806 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:52.701268 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:52.701985 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:52.701993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:52.701998 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:52.702123 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:52.702539 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:52.702544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:52.702547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:52.702814 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:52.703035 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:52.703733 (+ 698us) server_negotiation.cc:299] Negotiation successful
0928 02:52:52.703952 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":239,"threads_started":1}
I20260928 02:52:52.705309 2723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:52.687570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:35383)
0928 02:52:52.688293 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:52.688335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:52.689968 (+ 1633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:52.690698 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:52.690707 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:52.691291 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:52.691849 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:52.691867 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.693058 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.693062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:52.694457 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:52.694475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:52.695436 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:52.695444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:52.695737 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:52.696423 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:52.696451 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:52.698547 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:52.701412 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:52.701422 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:52.701441 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:52.701864 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:52.702230 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:52.702235 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:52.702239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:52.702424 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:52.703001 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:52.703011 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:52.703367 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:52:52.703657 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 02:52:52.704203 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":274,"threads_started":1}
I20260928 02:52:52.706924 2720 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:52.707263 2720 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:52.708176 2720 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
I20260928 02:52:52.709548 2263 ts_manager.cc:194] Registered new tserver with Master: 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:52.710320 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:35383
I20260928 02:52:52.720649 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.229.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.253.keytab.
Entry for principal oryx/127.27.229.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.253.keytab.
I20260928 02:52:52.771636 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40465
--webserver_interface=127.27.229.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.253.keytab
--principal=oryx/127.27.229.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:42995,127.27.229.253:40465
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:52:53.110318 2726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:53.110631 2726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:53.110723 2726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:53.118287 2726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:53.118414 2726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:53.118460 2726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:53.118510 2726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:53.118553 2726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:53.129246 2726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:42995,127.27.229.253:40465
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.253.keytab
--principal=oryx/127.27.229.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.229.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
W20260928 02:52:53.129606 2726 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.229.254:42995,127.27.229.253:40465'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 02:52:53.130658 2726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:53.132092 2726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:53.144362 2732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.144479 2731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.144412 2734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:53.146366 2726 server_base.cc:1034] running on GCE node
I20260928 02:52:53.147359 2726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:53.148854 2726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:53.150117 2726 hybrid_clock.cc:648] HybridClock initialized: now 1790563973150034 us; error 62 us; skew 500 ppm
I20260928 02:52:53.150462 2726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:53.153781 2726 init.cc:377] Logged in from keytab as oryx/127.27.229.253@KRBTEST.COM (short username oryx)
I20260928 02:52:53.155411 2726 webserver.cc:460] Webserver started at http://127.27.229.253:37493/ using document root <none> and password file <none>
I20260928 02:52:53.156183 2726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:53.156306 2726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:53.156615 2726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:53.159811 2726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/instance:
uuid: "309a187164ac4f27ae4a5e89274145b2"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "8f99659006ae22677c1461c42bc20b10"
server_key_iv: "036ec45a351943d1e7c6f1b6369c28c1"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.160596 2726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal/instance:
uuid: "309a187164ac4f27ae4a5e89274145b2"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "8f99659006ae22677c1461c42bc20b10"
server_key_iv: "036ec45a351943d1e7c6f1b6369c28c1"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.166397 2726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:52:53.170022 2741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:53.171468 2726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:53.171653 2726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "309a187164ac4f27ae4a5e89274145b2"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "8f99659006ae22677c1461c42bc20b10"
server_key_iv: "036ec45a351943d1e7c6f1b6369c28c1"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.171873 2726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:53.196410 2586 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
I20260928 02:52:53.202538 2726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:53.207161 2726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:53.207441 2726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:53.226452 2726 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.253:40465
I20260928 02:52:53.226471 2792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.253:40465 every 8 connection(s)
I20260928 02:52:53.227988 2726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
I20260928 02:52:53.231854 2793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:53.233539 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2726
I20260928 02:52:53.241721 2793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.253@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:53.260811 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.235853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57676 (local address 127.27.229.253:40465)
0928 02:52:53.236695 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.236710 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.236767 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:52:53.238455 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.238492 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.239111 (+ 619us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.239594 (+ 483us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.240964 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.243192 (+ 2228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.245568 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.246819 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.247020 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.251790 (+ 4770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.251799 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.251811 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.251859 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.255221 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.256272 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.256285 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.256297 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.256497 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.256923 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.256935 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.256937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.257723 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.257939 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.258613 (+ 674us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.259136 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":362,"threads_started":1}
I20260928 02:52:53.262144 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.244852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45416)
0928 02:52:53.245774 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.245832 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.246022 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.246656 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.246662 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.247625 (+ 963us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.247954 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.247975 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.249023 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.249028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.250513 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.250524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.251468 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.251475 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.251673 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.252387 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.252417 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.255157 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.259122 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.259135 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.259144 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.259593 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.260010 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.260014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.260016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.260311 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.261219 (+ 908us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.261227 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.261382 (+ 155us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.261568 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.261817 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":377,"threads_started":1}
I20260928 02:52:53.262336 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.244893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.27.229.254:42995)
0928 02:52:53.246304 (+ 1411us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.246310 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.246328 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:52:53.246399 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.246404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.246480 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.246598 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.248138 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.248872 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.250679 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.251198 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.251301 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.255385 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.255392 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.255396 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.255440 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.258967 (+ 3527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.259716 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.259722 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.259725 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.259867 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.260487 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.260493 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.260497 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.260937 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.261150 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.261701 (+ 551us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.261944 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1288,"thread_start_us":1186,"threads_started":1}
I20260928 02:52:53.264298 2793 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.253@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
I20260928 02:52:53.277943 2797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.265111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57686)
0928 02:52:53.265328 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.265378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.265505 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.265800 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.265804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.266190 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.266356 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.266364 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.267432 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.267437 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.268268 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.268278 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.268938 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.268943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.269152 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.269741 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.269757 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.271861 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.274965 (+ 3104us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.275070 (+ 105us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.275075 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.275740 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.276127 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.276132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.276136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.276289 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.276912 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.276919 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.277148 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.277390 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.277660 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":36}
I20260928 02:52:53.278187 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.265306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57686 (local address 127.27.229.253:40465)
0928 02:52:53.265469 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.265474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.265492 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:52:53.265640 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.265644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.265694 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.265787 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.266485 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.267299 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.268399 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.268801 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.268929 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.272060 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.272064 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.272068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.272096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.274766 (+ 2670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.275858 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.275862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.275864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.275981 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.276382 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.276385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.276387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.276671 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.276795 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.277490 (+ 695us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.277788 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":64}
I20260928 02:52:53.283129 2793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Bootstrap starting.
I20260928 02:52:53.286280 2793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:53.287444 2793 log.cc:824] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:53.291664 2793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: No bootstrap required, opened a new log
I20260928 02:52:53.298780 2793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } }
I20260928 02:52:53.299252 2793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:53.299345 2793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309a187164ac4f27ae4a5e89274145b2, State: Initialized, Role: FOLLOWER
I20260928 02:52:53.299991 2793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } }
I20260928 02:52:53.300910 2800 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } } }
I20260928 02:52:53.301317 2800 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:52:53.301980 2793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:52:53.312273 2811 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:52:53.312448 2811 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:52:53.315634 2726 master_runner.cc:428] Detected that this master 309a187164ac4f27ae4a5e89274145b2 is joining an existing cluster
I20260928 02:52:53.315801 2726 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.229.253:40465
WARNING: no policy specified for oryx/127.27.229.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.252@KRBTEST.COM" created.
I20260928 02:52:53.354075 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.320449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.27.229.254:42995)
0928 02:52:53.320634 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.320640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.333312 (+ 12672us) server_negotiation.cc:407] Connection header received
0928 02:52:53.333943 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.333952 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.334081 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.334172 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.335797 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.336947 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.342432 (+ 5485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.343188 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.343295 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.346094 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.346101 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.346106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.346160 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.350145 (+ 3985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.351462 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.351471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.351477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.351634 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.352098 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.352106 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.352111 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.352524 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.352658 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.353346 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.353616 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:52:53.355890 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.321190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45426)
0928 02:52:53.333150 (+ 11960us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.333177 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.333769 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.334409 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.334420 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.335182 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.335621 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.335636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.337149 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.337157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.340138 (+ 2981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.340159 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.343660 (+ 3501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.343667 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.343898 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.344650 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.344673 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.345860 (+ 1187us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.350328 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.350336 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.350342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.351309 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.351785 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.351792 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.351798 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.351969 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.352746 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.352753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.352973 (+ 220us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.354681 (+ 1708us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.355128 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11741,"mutex_wait_us":3,"thread_start_us":19259,"threads_started":1}
I20260928 02:52:53.361164 2817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.321222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57698)
0928 02:52:53.340857 (+ 19635us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.340886 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.341071 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.341595 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.341604 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.342344 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.342706 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.342721 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.344730 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.344733 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.346157 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.346170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.347637 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.347649 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.347856 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.348399 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.348428 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.349489 (+ 1061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.357974 (+ 8485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.357979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.357982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.358521 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.359077 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.359081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.359084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.359241 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.360033 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.360037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.360203 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.360364 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.360698 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19422,"thread_start_us":12096,"threads_started":1}
I20260928 02:52:53.362447 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.320746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.27.229.253:40465)
0928 02:52:53.321022 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.321030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.340962 (+ 19932us) server_negotiation.cc:407] Connection header received
0928 02:52:53.341230 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.341238 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.341324 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.341403 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.342894 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.344140 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.346364 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.347216 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.347327 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.349688 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.349694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.349699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.349733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.357828 (+ 8095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.358732 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.358737 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.358740 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.358941 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.359378 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.359385 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.359394 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.359806 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.359956 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.361908 (+ 1952us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.362121 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":63}
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.254@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
I20260928 02:52:53.382333 2796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.363820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57710 (local address 127.27.229.253:40465)
0928 02:52:53.363996 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.364003 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.367639 (+ 3636us) server_negotiation.cc:407] Connection header received
0928 02:52:53.368013 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.368021 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.368174 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.368271 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.370059 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.371101 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.372441 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.372903 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.373040 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.376603 (+ 3563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.376609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.376614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.376654 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.379517 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.380259 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.380268 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.380271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.380393 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.380795 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.380801 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.380804 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.381073 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.381221 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.381683 (+ 462us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.381922 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:52:53.382427 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.363323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57710)
0928 02:52:53.367483 (+ 4160us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.367532 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.367781 (+ 249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.368524 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.368532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.369545 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.369863 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.369878 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.371237 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.371243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.372258 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.372277 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.373066 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.373072 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.373272 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.373937 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.373959 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.376420 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.379670 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.379685 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.379694 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.380131 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.380501 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.380506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.380510 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.380681 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.381201 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.381209 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.381384 (+ 175us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.381574 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.381929 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3916,"thread_start_us":3783,"threads_started":1}
I20260928 02:52:53.384667 2263 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.229.253:40465: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "309a187164ac4f27ae4a5e89274145b2"
member_type: NON_VOTER
last_known_addr {
host: "127.27.229.253"
port: 40465
}
attrs {
promote: true
}
}
dest_uuid: "5a349fb854e649fcb24b3f62de8a1cc3"
cas_config_opid_index: -1
I20260928 02:52:53.385771 2263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } }
I20260928 02:52:53.392805 2824 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } }
I20260928 02:52:53.393965 2768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.252.keytab.
Entry for principal oryx/127.27.229.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.252.keytab.
I20260928 02:52:53.396889 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:35049
--webserver_interface=127.27.229.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.252.keytab
--principal=oryx/127.27.229.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:52:53.397964 2824 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.229.253:40465
I20260928 02:52:53.398028 2768 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 02:52:53.398455 2823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } }
I20260928 02:52:53.398762 2823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:53.398809 2726 master.cc:537] Master@127.27.229.253:40465 shutting down...
I20260928 02:52:53.401156 2824 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 02:52:53.404850 2248 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260928 02:52:53.407151 2248 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 -> Peer 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253:40465): Couldn't send request to peer 309a187164ac4f27ae4a5e89274145b2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:52:53.410959 2726 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:53.411386 2726 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:53.411588 2726 tablet_replica.cc:330] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: stopping tablet replica
I20260928 02:52:53.418479 2726 master.cc:552] Master@127.27.229.253:40465 shutdown complete.
I20260928 02:52:53.418669 2726 master_runner.cc:305] Clearing existing system tablet
I20260928 02:52:53.419263 2726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:53.419373 2726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:53.421752 2726 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260928 02:52:53.423278 2829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:53.423570 2726 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:53.423869 2726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:52:53.423949 2726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "309a187164ac4f27ae4a5e89274145b2"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "8f99659006ae22677c1461c42bc20b10"
server_key_iv: "036ec45a351943d1e7c6f1b6369c28c1"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.428516 2726 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:53.433286 2726 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:52:53.433480 2726 log.cc:1196] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 02:52:53.433871 2726 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Deleting consensus metadata
I20260928 02:52:53.434126 2726 master_runner.cc:315] Copying system tablet from 127.27.229.254:42995
I20260928 02:52:53.436646 2726 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: tablet copy: Beginning tablet copy session from remote peer at address 127.27.229.254:42995
I20260928 02:52:53.449445 2798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.437408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45430 (local address 127.27.229.254:42995)
0928 02:52:53.437616 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.437625 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.438105 (+ 480us) server_negotiation.cc:407] Connection header received
0928 02:52:53.438317 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.438323 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.438398 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.438499 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.439419 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.440150 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.441380 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.441902 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.442031 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.443952 (+ 1921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.443958 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.443963 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.444000 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.446877 (+ 2877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.447513 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.447519 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.447523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.447660 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.448033 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.448040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.448043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.448317 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.448471 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.448848 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.449060 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:52:53.449537 2835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.437365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45430)
0928 02:52:53.437991 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.438007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.438175 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.438522 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.438527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.439040 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.439282 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.439298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.440298 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.440303 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.441229 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.441240 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.442047 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.442053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.442256 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.442951 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.442969 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.443769 (+ 800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.447012 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.447018 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.447026 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.447388 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.447789 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.447794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.447798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.447927 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.448436 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.448442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.448606 (+ 164us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.448756 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.449088 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":391,"threads_started":1}
I20260928 02:52:53.450665 2283 tablet_copy_service.cc:140] P 5a349fb854e649fcb24b3f62de8a1cc3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 309a187164ac4f27ae4a5e89274145b2 ({username='oryx', principal='oryx/127.27.229.253@KRBTEST.COM'} at 127.0.0.1:45430)
I20260928 02:52:53.450987 2283 tablet_copy_service.cc:161] P 5a349fb854e649fcb24b3f62de8a1cc3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 309a187164ac4f27ae4a5e89274145b2 at {username='oryx', principal='oryx/127.27.229.253@KRBTEST.COM'} at 127.0.0.1:45430: session id = 309a187164ac4f27ae4a5e89274145b2-00000000000000000000000000000000
I20260928 02:52:53.454113 2283 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:52:53.456239 2726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:52:53.461264 2726 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: tablet copy: Starting download of 0 data blocks...
I20260928 02:52:53.461586 2726 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: tablet copy: Starting download of 1 WAL segments...
I20260928 02:52:53.464109 2726 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:52:53.467330 2283 tablet_copy_service.cc:342] P 5a349fb854e649fcb24b3f62de8a1cc3: Request end of tablet copy session 309a187164ac4f27ae4a5e89274145b2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.229.253@KRBTEST.COM'} at 127.0.0.1:45430
I20260928 02:52:53.467525 2283 tablet_copy_service.cc:433] P 5a349fb854e649fcb24b3f62de8a1cc3: ending tablet copy session 309a187164ac4f27ae4a5e89274145b2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 309a187164ac4f27ae4a5e89274145b2
I20260928 02:52:53.478243 2726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:53.481148 2836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.491261 2839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.491707 2837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:53.492409 2726 server_base.cc:1034] running on GCE node
I20260928 02:52:53.492763 2726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:53.493404 2726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:53.494621 2726 hybrid_clock.cc:648] HybridClock initialized: now 1790563973494549 us; error 36 us; skew 500 ppm
I20260928 02:52:53.494921 2726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:53.496750 2726 init.cc:377] Logged in from keytab as oryx/127.27.229.253@KRBTEST.COM (short username oryx)
I20260928 02:52:53.497999 2726 webserver.cc:460] Webserver started at http://127.27.229.253:40519/ using document root <none> and password file <none>
I20260928 02:52:53.498317 2726 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:53.498401 2726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:53.500761 2726 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:53.502341 2846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:53.502880 2726 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:52:53.503046 2726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "309a187164ac4f27ae4a5e89274145b2"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "8f99659006ae22677c1461c42bc20b10"
server_key_iv: "036ec45a351943d1e7c6f1b6369c28c1"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.503216 2726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:53.523905 2726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:53.527748 2726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:53.555770 2726 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.253:40465
I20260928 02:52:53.555747 2897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.253:40465 every 8 connection(s)
I20260928 02:52:53.556902 2726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
I20260928 02:52:53.564218 2898 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:52:53.566670 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Bootstrap starting.
I20260928 02:52:53.579458 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:52:53.579949 2898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: Bootstrap complete.
I20260928 02:52:53.580842 2898 raft_consensus.cc:348] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } }
I20260928 02:52:53.581070 2898 raft_consensus.cc:729] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 309a187164ac4f27ae4a5e89274145b2, State: Initialized, Role: LEARNER
I20260928 02:52:53.581331 2898 consensus_queue.cc:260] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } }
I20260928 02:52:53.582122 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } }
I20260928 02:52:53.582435 2898 sys_catalog.cc:564] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:53.582472 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:52:53.597139 2912 catalog_manager.cc:1259] Loaded cluster ID: b5f087ef201f423fbebdaba847989d27
I20260928 02:52:53.597270 2912 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: loading cluster ID for follower catalog manager: success
I20260928 02:52:53.599862 2912 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: acquiring CA information for follower catalog manager: success
I20260928 02:52:53.601670 2912 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:52:53.713160 2720 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
W20260928 02:52:53.821969 2827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:53.822381 2827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:53.822530 2827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:53.830013 2827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:52:53.830161 2827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:53.830256 2827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:53.830351 2827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:52:53.830422 2827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:52:53.841394 2827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.252.keytab
--principal=oryx/127.27.229.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:35049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.229.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:53.842844 2827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:53.843463 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.837361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57726)
0928 02:52:53.837553 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.837570 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.837670 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.838493 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.838498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.838534 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:53.838710 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.838719 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.840110 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.840115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.841356 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.841367 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.842603 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.842609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.842859 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.842906 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.843026 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.843169 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:52:53.843502 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.837546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57726 (local address 127.27.229.253:40465)
0928 02:52:53.838173 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.838178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.838194 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:52:53.838251 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.838254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.838325 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.838451 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:53.838870 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.839968 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.841496 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.842388 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.842603 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.842801 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.843051 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.843181 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":309,"threads_started":1}
I20260928 02:52:53.844426 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 02:52:53.844697 2873 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260928 02:52:53.845211 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a349fb854e649fcb24b3f62de8a1cc3. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } }
I20260928 02:52:53.845453 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: LEARNER
W20260928 02:52:53.857422 2918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.857301 2919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:53.857312 2921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:53.858664 2827 server_base.cc:1034] running on GCE node
I20260928 02:52:53.859560 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:53.861063 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:53.862317 2827 hybrid_clock.cc:648] HybridClock initialized: now 1790563973862270 us; error 49 us; skew 500 ppm
I20260928 02:52:53.862694 2827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:53.865878 2827 init.cc:377] Logged in from keytab as oryx/127.27.229.252@KRBTEST.COM (short username oryx)
I20260928 02:52:53.867411 2827 webserver.cc:460] Webserver started at http://127.27.229.252:40711/ using document root <none> and password file <none>
I20260928 02:52:53.868074 2827 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:53.868176 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:53.868412 2827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:52:53.871531 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/instance:
uuid: "a49999b294044db9946303f2140b8f9e"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "1f38ef5d70f2b884191c08fe5b35b230"
server_key_iv: "e11dbf58d36052e6eddf290871b7d29a"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.872252 2827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal/instance:
uuid: "a49999b294044db9946303f2140b8f9e"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "1f38ef5d70f2b884191c08fe5b35b230"
server_key_iv: "e11dbf58d36052e6eddf290871b7d29a"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.877933 2827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260928 02:52:53.883190 2928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:53.885015 2827 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:52:53.885219 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "a49999b294044db9946303f2140b8f9e"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "1f38ef5d70f2b884191c08fe5b35b230"
server_key_iv: "e11dbf58d36052e6eddf290871b7d29a"
server_key_version: "encryptionkey@0"
I20260928 02:52:53.885429 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:53.918284 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:53.922823 2827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:53.923243 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:53.942313 2827 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.252:35049
I20260928 02:52:53.942348 2979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.252:35049 every 8 connection(s)
I20260928 02:52:53.944033 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
I20260928 02:52:53.948411 2980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:53.951514 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 2827
I20260928 02:52:53.957231 2980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.252@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:52:53.976948 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.960360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.27.229.254:42995)
0928 02:52:53.961599 (+ 1239us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.961605 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.961623 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:52:53.961676 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.961682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.961750 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.961855 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.963396 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.964187 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.966180 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.966567 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.966700 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.971420 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.971425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.971428 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.971465 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.974068 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.974810 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.974817 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.974821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.974978 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.975385 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.975393 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.975398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.975668 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.975816 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.976336 (+ 520us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.976562 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":1024,"threads_started":1}
I20260928 02:52:53.978009 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.953990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.27.229.252:35049)
0928 02:52:53.955029 (+ 1039us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.955043 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.955092 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:52:53.957079 (+ 1987us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.957104 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.957581 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.958052 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.960089 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.962125 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.963982 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.964822 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.964953 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.970973 (+ 6020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.970982 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.970995 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.971032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.974020 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.974791 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.974798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.974800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.975010 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.975429 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.975455 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.975458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.975967 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.976289 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.976646 (+ 357us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.976892 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":391,"threads_started":1}
I20260928 02:52:53.978008 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.960097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45432)
0928 02:52:53.960927 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.960977 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.961188 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.961934 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.961939 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.962867 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.963235 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.963249 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.964343 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.964347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.965979 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.965995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.966875 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.966888 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.967131 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.967998 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.968033 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.971206 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.974228 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.974241 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.974254 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.974691 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.975085 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.975091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.975095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.975250 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.975793 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.975801 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.975998 (+ 197us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.976272 (+ 274us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.976779 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":411,"threads_started":1}
I20260928 02:52:53.979743 2980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:52:53 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.252@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
I20260928 02:52:53.994019 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.980446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57734)
0928 02:52:53.980643 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.980666 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.980772 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.981105 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.981108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.981413 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.981547 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.981575 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.982590 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.982595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.983837 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.983858 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.984599 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.984603 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.984761 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.985349 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:53.985368 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:53.987616 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:53.990974 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.990979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.990981 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.991435 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.992261 (+ 826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:53.992265 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:53.992268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:53.992423 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:53.993200 (+ 777us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:53.993204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:53.993333 (+ 129us) client_negotiation.cc:769] Sending connection context
0928 02:52:53.993507 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:52:53.993770 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260928 02:52:53.994210 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.980517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57734 (local address 127.27.229.253:40465)
0928 02:52:53.980695 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.980699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.980734 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:52:53.980896 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.980901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.980982 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.981062 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.981689 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.982452 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.984020 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.984433 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.984539 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:53.987846 (+ 3307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:53.987851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:53.987855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:53.987887 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:53.990755 (+ 2868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.991741 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.991750 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.991754 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.992033 (+ 279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:53.992599 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:53.992609 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:53.992612 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:53.993019 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:53.993215 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:53.993608 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 02:52:53.993873 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:52:53.995587 2980 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563973, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.252@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
I20260928 02:52:54.010567 2984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.996403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.0.0.1:45576)
0928 02:52:53.996606 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:53.996625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:53.996723 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:53.997101 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:53.997104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:53.997432 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:53.997579 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.997587 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.998960 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.998964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:53.999805 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:53.999815 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.000712 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.000716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.000874 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.001512 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.001530 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.003600 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.007594 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.007599 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.007603 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.008239 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.008804 (+ 565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.008809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.008811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.008994 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.009681 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.009685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.009850 (+ 165us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.009979 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.010220 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 02:52:54.010600 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:53.996641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.27.229.252:35049)
0928 02:52:53.996808 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:53.996814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:53.996831 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:52:53.996876 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:53.996881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:53.996946 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:53.997043 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:53.997701 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:53.998761 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:53.999934 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.000559 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.000690 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.003783 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.003791 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.003796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.003830 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.007348 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.008439 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.008446 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.008450 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.008628 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.009108 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.009112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.009116 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.009485 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.009615 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.010132 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.010345 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":55}
I20260928 02:52:54.015501 2980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Bootstrap starting.
I20260928 02:52:54.018684 2980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:54.019914 2980 log.cc:824] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:54.022756 2980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: No bootstrap required, opened a new log
I20260928 02:52:54.030501 2980 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } }
I20260928 02:52:54.031070 2980 raft_consensus.cc:374] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:54.031201 2980 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49999b294044db9946303f2140b8f9e, State: Initialized, Role: FOLLOWER
I20260928 02:52:54.031868 2980 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } }
I20260928 02:52:54.032830 2987 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } } }
I20260928 02:52:54.033123 2987 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:52:54.033941 2980 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:52:54.044301 2998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:52:54.044489 2998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:52:54.049566 2827 master_runner.cc:428] Detected that this master a49999b294044db9946303f2140b8f9e is joining an existing cluster
I20260928 02:52:54.049731 2827 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.229.252:35049
I20260928 02:52:54.078514 2916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.054274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.27.229.253:40465)
0928 02:52:54.054562 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.054566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.059812 (+ 5246us) server_negotiation.cc:407] Connection header received
0928 02:52:54.060174 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.060184 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.060291 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.060366 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.062942 (+ 2576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.064099 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.066398 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.067094 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.067190 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.070056 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.070065 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.070072 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.070112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.073681 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076135 (+ 2454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.076140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.076144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.076305 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076798 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.076805 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.076811 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.077179 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.077314 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.077909 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.078176 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260928 02:52:54.079159 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.053983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.27.229.252:35049)
0928 02:52:54.054178 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.054208 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.062169 (+ 7961us) server_negotiation.cc:407] Connection header received
0928 02:52:54.062764 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.062770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.062883 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.062991 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.064332 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.065560 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.067696 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.068468 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.068583 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.071074 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.071080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.071084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.071138 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.075435 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076282 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.076286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.076290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.076411 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076810 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.076814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.076818 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.077235 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.077382 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.077879 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.078133 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41,"spinlock_wait_cycles":44288}
I20260928 02:52:54.079814 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.053855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.27.229.254:42995)
0928 02:52:54.054051 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.054057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.062919 (+ 8862us) server_negotiation.cc:407] Connection header received
0928 02:52:54.064801 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.064807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.064893 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.064960 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.066459 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.067558 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.069699 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.070377 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.070472 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.073091 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.073099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.073105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.073154 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.076767 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.077489 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.077494 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.077498 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.077655 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.078133 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.078139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.078142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.078471 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.078611 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.079234 (+ 623us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.079464 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:52:54.080066 3006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.054073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57740)
0928 02:52:54.059617 (+ 5544us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.059659 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.059979 (+ 320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.060618 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.060622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.061484 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.061990 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.062009 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.064321 (+ 2312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.064326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.066207 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.066222 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.067488 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.067493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.067755 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.068600 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.068624 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.069852 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.073846 (+ 3994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.073852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.073857 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.075971 (+ 2114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.076460 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.076469 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.076656 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.077394 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.077401 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.077631 (+ 230us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.078956 (+ 1325us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.079445 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5315,"mutex_wait_us":3,"spinlock_wait_cycles":6912,"thread_start_us":8268,"threads_started":1}
I20260928 02:52:54.080705 3005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.054948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.0.0.1:45592)
0928 02:52:54.062039 (+ 7091us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.062060 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.062619 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.063163 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.063172 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.063839 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.064177 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.064191 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.065796 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.065801 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.067541 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.067554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.068907 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.068913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.069102 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.069764 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.069785 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.070905 (+ 1120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.075581 (+ 4676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.075586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.075590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.076159 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.076542 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076546 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.076551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.076696 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.077444 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.077448 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.077615 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.079536 (+ 1921us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.079829 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6897,"mutex_wait_us":27,"spinlock_wait_cycles":32000,"thread_start_us":3884,"threads_started":1}
I20260928 02:52:54.081081 3004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.055056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45444)
0928 02:52:54.062764 (+ 7708us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.062781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.064610 (+ 1829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.065148 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.065153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.065837 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.066303 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.066314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.067727 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.067732 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.069535 (+ 1803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.069547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.070802 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.070810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.071062 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.071717 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.071738 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.072865 (+ 1127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.076902 (+ 4037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.076907 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.076910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.077364 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.077810 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.077814 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.077817 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.077999 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.078719 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.078723 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.078963 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.080068 (+ 1105us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.080411 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7563,"mutex_wait_us":32,"spinlock_wait_cycles":3840,"thread_start_us":7131,"threads_started":1}
I20260928 02:52:54.084126 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2324
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.254@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.193@KRBTEST.COM
I20260928 02:52:54.101559 2983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.086975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.27.229.252:35049)
0928 02:52:54.087141 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.087146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.087167 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:52:54.087304 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.087310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.087453 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.088469 (+ 1016us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.088669 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.089454 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.091181 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.092027 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.092160 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.095612 (+ 3452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.095618 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.095621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.095652 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.098884 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.099556 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.099561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.099565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.099683 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.100053 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.100058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.100061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.100454 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.100617 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.100981 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.101220 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:52:54.101577 2821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.086910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.0.0.1:45608)
0928 02:52:54.087064 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.087082 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.087175 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.087649 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.087654 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.088140 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.088345 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.088360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.089573 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.089579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.090998 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.091024 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.092192 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.092199 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.092374 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.093104 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.093130 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.095410 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.099053 (+ 3643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.099059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.099063 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.099423 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.099791 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.099799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.099803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.099933 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.100576 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.100581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.100741 (+ 160us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.100874 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.101162 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 02:52:54.103379 2263 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.229.252:35049: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a49999b294044db9946303f2140b8f9e"
member_type: NON_VOTER
last_known_addr {
host: "127.27.229.252"
port: 35049
}
attrs {
promote: true
}
}
dest_uuid: "5a349fb854e649fcb24b3f62de8a1cc3"
cas_config_opid_index: 5
I20260928 02:52:54.104391 2263 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } }
I20260928 02:52:54.106320 2823 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a49999b294044db9946303f2140b8f9e (127.27.229.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.106897 3007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.085272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.27.229.193:41133)
0928 02:52:54.085985 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.085992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.086018 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:52:54.086074 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.086080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.086290 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.086472 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.088119 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.089759 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.091563 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.092323 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.093026 (+ 703us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.098044 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.098063 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.098086 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.098136 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.102205 (+ 4069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.103240 (+ 1035us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.103250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.103254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.103415 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.103906 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.103912 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.103915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.104399 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.105893 (+ 1494us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.106100 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.106402 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":387,"threads_started":1}
I20260928 02:52:54.111773 2873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:52:54.112558 2447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:54.113950 2955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:54.120033 3008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 02:52:54.121058 2823 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.229.252:35049
I20260928 02:52:54.121713 2827 master.cc:537] Master@127.27.229.252:35049 shutting down...
I20260928 02:52:54.121583 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.121960 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:54.123399 2955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 02:52:54.124475 3008 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:54.137253 2873 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a49999b294044db9946303f2140b8f9e (127.27.229.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
W20260928 02:52:54.143672 2250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 -> Peer a49999b294044db9946303f2140b8f9e (127.27.229.252:35049): Couldn't send request to peer a49999b294044db9946303f2140b8f9e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:52:54.144449 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: NON_VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: true } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.144762 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: LEARNER
I20260928 02:52:54.145328 3009 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: attempting to promote NON_VOTER 309a187164ac4f27ae4a5e89274145b2 to VOTER
I20260928 02:52:54.146070 3009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } }
I20260928 02:52:54.148460 2873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 LEARNER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 02:52:54.149250 3009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260928 02:52:54.150221 2250 consensus_peers.cc:487] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 -> Peer a49999b294044db9946303f2140b8f9e (127.27.229.252:35049): Couldn't send request to peer a49999b294044db9946303f2140b8f9e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 02:52:54.151621 3008 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.154704 2823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.154979 2823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:54.154750 3008 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.155313 3008 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:54.155349 2873 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.158443 2901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.158710 2901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:52:54.175410 2934 tablet.cc:2369] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:54.177510 2827 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:54.177923 2827 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:54.178138 2827 tablet_replica.cc:330] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: stopping tablet replica
I20260928 02:52:54.184297 2827 master.cc:552] Master@127.27.229.252:35049 shutdown complete.
I20260928 02:52:54.184398 2827 master_runner.cc:305] Clearing existing system tablet
I20260928 02:52:54.184751 2827 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:54.184872 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:54.187198 2827 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260928 02:52:54.188663 3015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:54.189051 2827 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:54.189302 2827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 02:52:54.189399 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "a49999b294044db9946303f2140b8f9e"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "1f38ef5d70f2b884191c08fe5b35b230"
server_key_iv: "e11dbf58d36052e6eddf290871b7d29a"
server_key_version: "encryptionkey@0"
I20260928 02:52:54.193217 2827 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:54.198176 2827 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 02:52:54.198369 2827 log.cc:1196] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 02:52:54.198778 2827 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Deleting consensus metadata
I20260928 02:52:54.199096 2827 master_runner.cc:315] Copying system tablet from 127.27.229.254:42995
I20260928 02:52:54.203137 2827 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: tablet copy: Beginning tablet copy session from remote peer at address 127.27.229.254:42995
I20260928 02:52:54.215963 2985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.204029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45446 (local address 127.27.229.254:42995)
0928 02:52:54.204254 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.204262 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.204966 (+ 704us) server_negotiation.cc:407] Connection header received
0928 02:52:54.205173 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.205178 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.205253 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.205356 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.206188 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.207021 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.208217 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.208631 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.208772 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.210629 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.210637 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.210642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.210677 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.213380 (+ 2703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.214015 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.214025 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.214029 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.214147 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.214521 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.214528 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.214532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.214808 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.215024 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.215380 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.215589 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 02:52:54.216044 3021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.204029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45446)
0928 02:52:54.204851 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.204871 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.205041 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.205387 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.205392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.205832 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.206032 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.206040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.207162 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.207166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.208082 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.208093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.208792 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.208798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.208973 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.209687 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.209705 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.210424 (+ 719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.213525 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.213530 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.213534 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.213886 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.214259 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.214264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.214267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.214413 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.214977 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.214982 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.215148 (+ 166us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.215282 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.215606 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":318,"threads_started":1}
I20260928 02:52:54.216862 2283 tablet_copy_service.cc:140] P 5a349fb854e649fcb24b3f62de8a1cc3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a49999b294044db9946303f2140b8f9e ({username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:45446)
I20260928 02:52:54.217033 2283 tablet_copy_service.cc:161] P 5a349fb854e649fcb24b3f62de8a1cc3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a49999b294044db9946303f2140b8f9e at {username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:45446: session id = a49999b294044db9946303f2140b8f9e-00000000000000000000000000000000
I20260928 02:52:54.219386 2283 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 02:52:54.220968 2827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 02:52:54.225742 2827 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: tablet copy: Starting download of 0 data blocks...
I20260928 02:52:54.226048 2827 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: tablet copy: Starting download of 1 WAL segments...
I20260928 02:52:54.228426 2827 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 02:52:54.231907 2283 tablet_copy_service.cc:342] P 5a349fb854e649fcb24b3f62de8a1cc3: Request end of tablet copy session a49999b294044db9946303f2140b8f9e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:45446
I20260928 02:52:54.232053 2283 tablet_copy_service.cc:433] P 5a349fb854e649fcb24b3f62de8a1cc3: ending tablet copy session a49999b294044db9946303f2140b8f9e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a49999b294044db9946303f2140b8f9e
I20260928 02:52:54.243124 2827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:54.245967 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:54.247130 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:54.259392 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:54.259898 2827 server_base.cc:1034] running on GCE node
I20260928 02:52:54.260243 2827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:54.261392 2827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:54.262619 2827 hybrid_clock.cc:648] HybridClock initialized: now 1790563974262555 us; error 45 us; skew 500 ppm
I20260928 02:52:54.262933 2827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563974, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:54.264776 2827 init.cc:377] Logged in from keytab as oryx/127.27.229.252@KRBTEST.COM (short username oryx)
I20260928 02:52:54.266068 2827 webserver.cc:460] Webserver started at http://127.27.229.252:33479/ using document root <none> and password file <none>
I20260928 02:52:54.266390 2827 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:54.266495 2827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:54.268904 2827 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 02:52:54.270571 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:54.271008 2827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260928 02:52:54.271139 2827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "a49999b294044db9946303f2140b8f9e"
format_stamp: "Formatted at 2026-09-28 02:52:53 on dist-test-slave-s1kg"
server_key: "1f38ef5d70f2b884191c08fe5b35b230"
server_key_iv: "e11dbf58d36052e6eddf290871b7d29a"
server_key_version: "encryptionkey@0"
I20260928 02:52:54.271325 2827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:54.280478 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2324
I20260928 02:52:54.291026 2827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:54.294734 2827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:54.304414 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:41133
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:42995
--webserver_port=34553
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:52:54.327406 2827 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.252:35049
I20260928 02:52:54.327461 3084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.252:35049 every 8 connection(s)
I20260928 02:52:54.328529 2827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
I20260928 02:52:54.335767 3085 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:52:54.341460 3085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Bootstrap starting.
I20260928 02:52:54.364694 3085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:52:54.365224 3085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Bootstrap complete.
I20260928 02:52:54.366092 3085 raft_consensus.cc:348] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } }
I20260928 02:52:54.366335 3085 raft_consensus.cc:729] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a49999b294044db9946303f2140b8f9e, State: Initialized, Role: LEARNER
I20260928 02:52:54.366577 3085 consensus_queue.cc:260] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } }
I20260928 02:52:54.371831 3088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.372365 3085 sys_catalog.cc:564] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:52:54.372364 3088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: LEARNER
I20260928 02:52:54.390337 3099 catalog_manager.cc:1259] Loaded cluster ID: b5f087ef201f423fbebdaba847989d27
I20260928 02:52:54.390462 3099 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: loading cluster ID for follower catalog manager: success
I20260928 02:52:54.393924 3099 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: acquiring CA information for follower catalog manager: success
I20260928 02:52:54.396440 3099 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:52:54.684621 3053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:54.684926 3053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:54.685055 3053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:54.692780 3053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:54.692906 3053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:54.693087 3053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:52:54.703621 3053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:41133
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=34553
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:54.705121 3053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:54.706493 3053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:54.721138 3104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:54.721117 3107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:54.721088 3105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:54.721766 3053 server_base.cc:1034] running on GCE node
I20260928 02:52:54.722478 3053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:54.723555 3053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:54.724802 3053 hybrid_clock.cc:648] HybridClock initialized: now 1790563974724756 us; error 64 us; skew 500 ppm
I20260928 02:52:54.725167 3053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563974, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:54.729547 3053 init.cc:377] Logged in from keytab as oryx/127.27.229.193@KRBTEST.COM (short username oryx)
I20260928 02:52:54.731339 3053 webserver.cc:460] Webserver started at http://127.27.229.193:34553/ using document root <none> and password file <none>
I20260928 02:52:54.732084 3053 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:54.732225 3053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:54.738663 3053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 02:52:54.742043 3114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:54.743678 3053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:52:54.743848 3053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "07c3a929c9984276a84754db1d1450a4"
format_stamp: "Formatted at 2026-09-28 02:52:51 on dist-test-slave-s1kg"
server_key: "f3b6cfe880cfe1fd64f1aefb0ff2bdd5"
server_key_iv: "001b7d366d845459a968cb2cf6ad3334"
server_key_version: "encryptionkey@0"
I20260928 02:52:54.745033 3053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:54.774894 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.768480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45618 (local address 127.27.229.252:35049)
0928 02:52:54.769126 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.769131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.769146 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:52:54.769205 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.769208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.769274 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.769408 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:54.769781 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.770935 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.772817 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.773830 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.774011 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.774208 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.774401 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.774530 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":366,"threads_started":1}
I20260928 02:52:54.774878 3119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.768320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.0.0.1:45618)
0928 02:52:54.768789 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.768811 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.768966 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.769425 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.769430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.769456 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:54.769642 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.769651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.771085 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.771093 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.772674 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.772688 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.774011 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.774018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.774214 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.774256 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.774379 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.774530 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":255,"threads_started":1}
I20260928 02:52:54.775099 3053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:54.775884 3060 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 02:52:54.776602 3088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a349fb854e649fcb24b3f62de8a1cc3. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:54.776872 3088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: LEARNER
I20260928 02:52:54.780200 3053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:54.780520 3053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:54.781526 3053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:54.783284 3053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:54.783392 3053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:54.783495 3053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:54.783576 3053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:54.817584 3053 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:41133
I20260928 02:52:54.817677 3228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:41133 every 8 connection(s)
I20260928 02:52:54.819106 3053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:52:54.824524 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3053
I20260928 02:52:54.824882 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2458
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.194@KRBTEST.COM
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563974, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563974, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
Sep 28 02:52:54 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563974, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
I20260928 02:52:54.862474 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.824451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:41815 (local address 127.27.229.252:35049)
0928 02:52:54.824676 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.824684 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.842661 (+ 17977us) server_negotiation.cc:407] Connection header received
0928 02:52:54.842934 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.842941 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.843062 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.843139 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.844332 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.845706 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.848139 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.848837 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.848939 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.854858 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.854866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.854870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.854911 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.858321 (+ 3410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.859552 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.859557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.859560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.859735 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.860249 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.860254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.860257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.860612 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.860746 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.861846 (+ 1100us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.862097 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:52:54.862550 3236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.823451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39769 (local address 127.27.229.254:42995)
0928 02:52:54.837430 (+ 13979us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.837439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.837463 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:52:54.837619 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.837628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.837744 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.837923 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.842882 (+ 4959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.844163 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.847328 (+ 3165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.847975 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.848098 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.854494 (+ 6396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.854501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.854506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.854553 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.858223 (+ 3670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.859491 (+ 1268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.859498 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.859502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.859665 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.860222 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.860229 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.860235 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.860621 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.860763 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.861864 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.862139 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13821,"thread_start_us":13704,"threads_started":1}
I20260928 02:52:54.862949 3235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.823113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:57725 (local address 127.27.229.253:40465)
0928 02:52:54.837016 (+ 13903us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.837024 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.838820 (+ 1796us) server_negotiation.cc:407] Connection header received
0928 02:52:54.839044 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.839051 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.839160 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.839280 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.842908 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.844096 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.847234 (+ 3138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.847886 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.847999 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.855366 (+ 7367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.855372 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.855378 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.855461 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.859970 (+ 4509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.860763 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.860769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.860773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.860915 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.861356 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.861361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.861365 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.861698 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.861820 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.862312 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.862542 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13739,"thread_start_us":13611,"threads_started":1}
I20260928 02:52:54.863770 3239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.825834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.27.229.194:39513)
0928 02:52:54.836379 (+ 10545us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:54.836392 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:54.839598 (+ 3206us) server_negotiation.cc:407] Connection header received
0928 02:52:54.840264 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:54.840274 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:54.840502 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:54.840599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:54.842304 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.844104 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.846098 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.847048 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.847178 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.853830 (+ 6652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:54.853847 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:54.853867 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:54.853915 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:54.859375 (+ 5460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.860406 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.860415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.860418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.860572 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:54.861030 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:54.861034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:54.861037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:54.861447 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:54.861768 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:54.863041 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0928 02:52:54.863321 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10299,"thread_start_us":10560,"threads_started":1}
I20260928 02:52:54.865195 2581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:54.865458 3237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.822671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.27.229.193:57725)
0928 02:52:54.838672 (+ 16001us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.838704 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.838873 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.839468 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.839474 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.840076 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.842653 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.842671 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.844304 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.844308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.847064 (+ 2756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.847079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.848532 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.848541 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.848950 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.850082 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.850110 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.855117 (+ 5007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.860118 (+ 5001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.860136 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.860143 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.860624 (+ 481us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.861049 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.861053 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.861056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.861225 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.861894 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.861898 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.862030 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.863219 (+ 1189us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.863993 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15806,"mutex_wait_us":624,"spinlock_wait_cycles":103680,"thread_start_us":17178,"threads_started":1}
I20260928 02:52:54.865525 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.822671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:39769)
0928 02:52:54.823780 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.823844 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.833075 (+ 9231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.838471 (+ 5396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.838486 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.839532 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.842621 (+ 3089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.842653 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.844342 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.844347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.847132 (+ 2785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.847157 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.848533 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.848540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.848913 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.849788 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.849821 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.854185 (+ 4364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.858417 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.858430 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.858455 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.859330 (+ 875us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.859823 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.859828 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.859832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.860094 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.860851 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.860856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.861406 (+ 550us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.863219 (+ 1813us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.863884 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":147,"spinlock_wait_cycles":6144,"thread_start_us":465,"threads_started":1}
I20260928 02:52:54.866165 3234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:54.823991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.27.229.193:41815)
0928 02:52:54.842550 (+ 18559us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:54.842573 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:54.842763 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:54.843350 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:54.843355 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:54.843892 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:54.844163 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.844174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.846337 (+ 2163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.846344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.847975 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:54.847988 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:54.849240 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:54.849247 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:54.849437 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:54.850193 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:54.850217 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:54.854573 (+ 4356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:54.858487 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.858493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.858497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.859404 (+ 907us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.859898 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:54.859903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:54.859906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:54.860090 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:54.860838 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:54.860847 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:54.861403 (+ 556us) client_negotiation.cc:769] Sending connection context
0928 02:52:54.863373 (+ 1970us) client_negotiation.cc:240] Negotiation successful
0928 02:52:54.864236 (+ 863us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18401,"mutex_wait_us":528,"spinlock_wait_cycles":11008,"thread_start_us":14346,"threads_started":1}
I20260928 02:52:54.867139 3230 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:54.867290 3231 heartbeater.cc:347] Connected to a master server at 127.27.229.253:40465
I20260928 02:52:54.867511 3230 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:54.867517 3231 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:54.868116 3229 heartbeater.cc:347] Connected to a master server at 127.27.229.252:35049
I20260928 02:52:54.868333 3229 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:54.868713 3231 heartbeater.cc:511] Master 127.27.229.253:40465 requested a full tablet report, sending...
I20260928 02:52:54.868907 3229 heartbeater.cc:511] Master 127.27.229.252:35049 requested a full tablet report, sending...
I20260928 02:52:54.869350 3230 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
I20260928 02:52:54.871351 3049 ts_manager.cc:194] Registered new tserver with Master: 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133)
I20260928 02:52:54.871351 2863 ts_manager.cc:194] Registered new tserver with Master: 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133)
I20260928 02:52:54.872304 2263 ts_manager.cc:194] Re-registered known tserver with Master: 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133)
I20260928 02:52:54.873466 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:39769
I20260928 02:52:54.957211 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2458
I20260928 02:52:54.975059 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:39513
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:42995
--webserver_port=39679
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:52:55.182780 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: NON_VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: true } } }
I20260928 02:52:55.183279 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:55.184911 3009 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: attempting to promote NON_VOTER a49999b294044db9946303f2140b8f9e to VOTER
I20260928 02:52:55.185495 3009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } }
I20260928 02:52:55.188002 3060 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 LEARNER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:52:55.188304 2873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Refusing update from remote peer 5a349fb854e649fcb24b3f62de8a1cc3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260928 02:52:55.188654 3243 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:52:55.189072 3009 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260928 02:52:55.191717 3243 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a49999b294044db9946303f2140b8f9e (127.27.229.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.192445 2873 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a49999b294044db9946303f2140b8f9e (127.27.229.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.195568 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.195859 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:55.195792 3243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.196069 3243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:55.197070 3248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.197511 3248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:52:55.198376 3009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.198594 3009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:55.199200 3060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, a49999b294044db9946303f2140b8f9e (127.27.229.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.202243 3088 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:55.202718 3088 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:52:55.315274 3241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:55.315568 3241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:55.315728 3241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:55.323133 3241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:55.323253 3241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:55.323421 3241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:52:55.337707 3241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:39513
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=39679
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:55.339372 3241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:55.340966 3241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:55.355599 3254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:55.355628 3256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:55.356096 3253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:55.356565 3241 server_base.cc:1034] running on GCE node
I20260928 02:52:55.357255 3241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:55.358229 3241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:55.359480 3241 hybrid_clock.cc:648] HybridClock initialized: now 1790563975359432 us; error 30 us; skew 500 ppm
I20260928 02:52:55.359900 3241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:55.364602 3241 init.cc:377] Logged in from keytab as oryx/127.27.229.194@KRBTEST.COM (short username oryx)
I20260928 02:52:55.366729 3241 webserver.cc:460] Webserver started at http://127.27.229.194:39679/ using document root <none> and password file <none>
I20260928 02:52:55.367712 3241 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:55.367918 3241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:55.374959 3241 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.004s
I20260928 02:52:55.377877 3263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:55.379573 3241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:55.379731 3241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "2c6cf40e3b054e839524736121a2bf11"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "6882a6ae3de3b6d49adfafaae430e21d"
server_key_iv: "83ec06f8448098e3529239c9d2847d7a"
server_key_version: "encryptionkey@0"
I20260928 02:52:55.380726 3241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:55.396059 3241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:55.400524 3241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:55.400780 3241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:55.401718 3241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:55.403122 3241 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:55.403209 3241 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:55.403328 3241 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:55.403398 3241 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:55.440939 3241 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:39513
I20260928 02:52:55.440989 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:39513 every 8 connection(s)
I20260928 02:52:55.442513 3241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:52:55.447762 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3241
I20260928 02:52:55.448055 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2592
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563970, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.195@KRBTEST.COM
I20260928 02:52:55.480908 3381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.445892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:38447 (local address 127.27.229.253:40465)
0928 02:52:55.452464 (+ 6572us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:55.452469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:55.458639 (+ 6170us) server_negotiation.cc:407] Connection header received
0928 02:52:55.458824 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:55.458850 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:55.458941 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:55.459034 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:55.462275 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.463362 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.465855 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.466464 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.466563 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.472664 (+ 6101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:55.472674 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:55.472680 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:55.472740 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:55.476509 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.477977 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.477985 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.477988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.478133 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.478706 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.478711 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.478714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.479104 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:55.479238 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:55.480274 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 02:52:55.480532 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6462,"thread_start_us":6366,"threads_started":1}
I20260928 02:52:55.480912 3384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.447856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:43837 (local address 127.27.229.254:42995)
0928 02:52:55.456249 (+ 8393us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:55.456257 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:55.457857 (+ 1600us) server_negotiation.cc:407] Connection header received
0928 02:52:55.458947 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:55.458956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:55.459074 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:55.459237 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:55.462286 (+ 3049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.463783 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.466454 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.467340 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.467464 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.472656 (+ 5192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:55.472667 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:55.472674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:55.472731 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:55.476856 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.478013 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.478019 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.478022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.478187 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.478708 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.478714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.478717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.479058 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:55.479176 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:55.480292 (+ 1116us) server_negotiation.cc:299] Negotiation successful
0928 02:52:55.480518 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8272,"thread_start_us":8180,"threads_started":1}
I20260928 02:52:55.483297 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.448298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.27.229.194:38447)
0928 02:52:55.458531 (+ 10233us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:55.458544 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:55.458683 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:55.459494 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:55.459506 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:55.460688 (+ 1182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:55.462007 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.462038 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.463568 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.463573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.465691 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.465706 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.466963 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.466975 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.467314 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.468028 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:55.468062 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:55.472326 (+ 4264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:55.476701 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.476711 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.476733 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.477815 (+ 1082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.478285 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.478289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.478293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.478556 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.479332 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:55.479336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:55.479799 (+ 463us) client_negotiation.cc:769] Sending connection context
0928 02:52:55.481171 (+ 1372us) client_negotiation.cc:240] Negotiation successful
0928 02:52:55.481892 (+ 721us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10146,"mutex_wait_us":399,"spinlock_wait_cycles":60288,"thread_start_us":6767,"threads_started":1}
I20260928 02:52:55.483399 3383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.447455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:43837)
0928 02:52:55.457758 (+ 10303us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:55.457782 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:55.457953 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:55.459497 (+ 1544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:55.459509 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:55.460692 (+ 1183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:55.462007 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.462040 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.464037 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.464045 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.466246 (+ 2201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.466263 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.467828 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.467834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.468147 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.468837 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:55.468869 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:55.472326 (+ 3457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:55.477011 (+ 4685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.477016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.477020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.477845 (+ 825us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.478334 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.478339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.478343 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.478559 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.479276 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:55.479284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:55.479797 (+ 513us) client_negotiation.cc:769] Sending connection context
0928 02:52:55.481121 (+ 1324us) client_negotiation.cc:240] Negotiation successful
0928 02:52:55.481733 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10159,"mutex_wait_us":147,"spinlock_wait_cycles":19712,"thread_start_us":9976,"threads_started":1}
I20260928 02:52:55.484156 3387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.449144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55266 (local address 127.27.229.195:43787)
0928 02:52:55.456766 (+ 7622us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:55.456774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:55.456805 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:52:55.456883 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:55.456892 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:55.457138 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:55.461133 (+ 3995us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:55.461536 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.463189 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.465129 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.465960 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.473181 (+ 7221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.474088 (+ 907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:55.474104 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:55.474135 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:55.474175 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:55.479171 (+ 4996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.480095 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.480117 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.480120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.480277 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.480748 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.480753 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.480757 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.481174 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:55.482882 (+ 1708us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:55.483062 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:52:55.483749 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7433,"thread_start_us":7272,"threads_started":1}
I20260928 02:52:55.485008 3378 heartbeater.cc:347] Connected to a master server at 127.27.229.253:40465
I20260928 02:52:55.485225 3377 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:55.485361 3377 heartbeater.cc:464] Registering TS with master...
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
I20260928 02:52:55.486320 3377 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
I20260928 02:52:55.485365 3378 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:55.487310 2715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:55.487795 3378 heartbeater.cc:511] Master 127.27.229.253:40465 requested a full tablet report, sending...
I20260928 02:52:55.488040 2263 ts_manager.cc:194] Re-registered known tserver with Master: 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:55.489298 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:43837
I20260928 02:52:55.490139 2863 ts_manager.cc:194] Registered new tserver with Master: 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:55.494254 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.445751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:59223 (local address 127.27.229.252:35049)
0928 02:52:55.476684 (+ 30933us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:55.476690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:55.476709 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:52:55.476766 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:55.476771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:55.476861 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:55.476977 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:55.478192 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.479338 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.481731 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.482449 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.483211 (+ 762us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.487329 (+ 4118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:55.487338 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:55.487346 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:55.487391 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:55.491088 (+ 3697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.492033 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.492038 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.492042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.492185 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:55.492668 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:55.492675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:55.492679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:55.493051 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:55.493219 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:55.493638 (+ 419us) server_negotiation.cc:299] Negotiation successful
0928 02:52:55.493852 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30781,"thread_start_us":30645,"threads_started":1}
I20260928 02:52:55.494280 3385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:55.446564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.27.229.194:59223)
0928 02:52:55.455411 (+ 8847us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:55.455455 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:55.457944 (+ 2489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:55.477001 (+ 19057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:55.477007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:55.477683 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:55.478012 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.478023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.479485 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.479490 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.481024 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:55.481038 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:55.482662 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:55.482669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:55.482877 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:55.484081 (+ 1204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:55.484112 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:55.487059 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:55.491255 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.491260 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.491264 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.491795 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.492320 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:55.492325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:55.492328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:55.492520 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:55.493218 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:55.493224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:55.493381 (+ 157us) client_negotiation.cc:769] Sending connection context
0928 02:52:55.493534 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 02:52:55.493867 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8560,"thread_start_us":11594,"threads_started":1}
I20260928 02:52:55.495404 3376 heartbeater.cc:347] Connected to a master server at 127.27.229.252:35049
I20260928 02:52:55.495554 3376 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:55.495949 3376 heartbeater.cc:511] Master 127.27.229.252:35049 requested a full tablet report, sending...
I20260928 02:52:55.497012 3049 ts_manager.cc:194] Registered new tserver with Master: 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:55.577474 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2592
I20260928 02:52:55.594321 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:43787
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:42995
--webserver_port=40269
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:52:55.875936 3230 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
W20260928 02:52:55.935932 3389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:52:55.936226 3389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:52:55.936386 3389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:52:55.944063 3389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:52:55.944185 3389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:52:55.944341 3389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:52:55.954533 3389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:43787
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=40269
--tserver_master_addrs=127.27.229.254:42995,127.27.229.253:40465,127.27.229.252:35049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:52:55.955971 3389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:52:55.957340 3389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:52:55.970647 3394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:55.970685 3395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:52:55.970709 3397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:52:55.971323 3389 server_base.cc:1034] running on GCE node
I20260928 02:52:55.972007 3389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:52:55.973140 3389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:52:55.974339 3389 hybrid_clock.cc:648] HybridClock initialized: now 1790563975974293 us; error 44 us; skew 500 ppm
I20260928 02:52:55.974701 3389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:52:55 dist-test-slave-s1kg krb5kdc[2218](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:52:55.977821 3389 init.cc:377] Logged in from keytab as oryx/127.27.229.195@KRBTEST.COM (short username oryx)
I20260928 02:52:55.979321 3389 webserver.cc:460] Webserver started at http://127.27.229.195:40269/ using document root <none> and password file <none>
I20260928 02:52:55.979964 3389 fs_manager.cc:359] Metadata directory not provided
I20260928 02:52:55.980072 3389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:52:55.986230 3389 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:52:55.989251 3404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:55.991035 3389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:52:55.991220 3389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "695ebf22b5784e60a9830411707b6531"
format_stamp: "Formatted at 2026-09-28 02:52:52 on dist-test-slave-s1kg"
server_key: "27b22beb5bab9b6e1c6194522f7b5a85"
server_key_iv: "841b4a0e0c8691357d80c15f44ed5ccf"
server_key_version: "encryptionkey@0"
I20260928 02:52:55.992271 3389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:52:56.014446 3389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:52:56.019394 3389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:52:56.019724 3389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:52:56.020816 3389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:52:56.022322 3389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:52:56.022414 3389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:56.022539 3389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:52:56.022621 3389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:52:56.058588 3389 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43787
I20260928 02:52:56.058807 3516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43787 every 8 connection(s)
I20260928 02:52:56.060141 3389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:52:56.067198 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3389
Sep 28 02:52:56 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
Sep 28 02:52:56 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.252@KRBTEST.COM
Sep 28 02:52:56 dist-test-slave-s1kg krb5kdc[2218](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563975, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.253@KRBTEST.COM
I20260928 02:52:56.132808 3525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.064698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46853 (local address 127.27.229.254:42995)
0928 02:52:56.092513 (+ 27815us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.092523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.096137 (+ 3614us) server_negotiation.cc:407] Connection header received
0928 02:52:56.098870 (+ 2733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.098877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.099006 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.099127 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.107268 (+ 8141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.108363 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.111318 (+ 2955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.111964 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.112082 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.119084 (+ 7002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.119093 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.119101 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.119145 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.127998 (+ 8853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.128820 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.128829 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.128836 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.129029 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.129621 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.129629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.129636 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.130106 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.130253 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.131733 (+ 1480us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.131984 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27605,"thread_start_us":27839,"threads_started":1}
I20260928 02:52:56.133675 3526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.064063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:46853)
0928 02:52:56.095929 (+ 31866us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.095996 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.098515 (+ 2519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.104613 (+ 6098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.104618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.106540 (+ 1922us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:56.106774 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.106785 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.109430 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.109435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.110932 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.110946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.112186 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.112205 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.113096 (+ 891us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.113810 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:56.113861 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:56.118516 (+ 4655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:56.128138 (+ 9622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.128143 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.128147 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.128693 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.129135 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.129141 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.129144 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.129503 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.130246 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:56.130255 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:56.131416 (+ 1161us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.131597 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.132208 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31501,"mutex_wait_us":1413,"thread_start_us":39617,"threads_started":1}
I20260928 02:52:56.134893 3521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.063394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:41837 (local address 127.27.229.253:40465)
0928 02:52:56.089351 (+ 25957us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.089357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.099494 (+ 10137us) server_negotiation.cc:407] Connection header received
0928 02:52:56.099807 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.099817 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.099942 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.100045 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.105181 (+ 5136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.106412 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.111122 (+ 4710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.111770 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.111869 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.122048 (+ 10179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.122055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.122059 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.122110 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.126893 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.128965 (+ 2072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.128972 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.128976 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.129149 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.129673 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.129678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.129680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.130067 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.130227 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.134216 (+ 3989us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.134471 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25826,"thread_start_us":26221,"threads_started":1}
I20260928 02:52:56.135845 3524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.064438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.27.229.195:41837)
0928 02:52:56.099366 (+ 34928us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.099389 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.099612 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.100873 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.100892 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.102234 (+ 1342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:56.103473 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.103520 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.106998 (+ 3478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.107003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.109278 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.109293 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.112696 (+ 3403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.112702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.112856 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.114014 (+ 1158us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:56.114057 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:56.121769 (+ 7712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:56.127195 (+ 5426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.127211 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.127242 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.128253 (+ 1011us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.129298 (+ 1045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.129303 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.129306 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.129506 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.130318 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:56.130323 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:56.130918 (+ 595us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.135074 (+ 4156us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.135396 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34755,"mutex_wait_us":1,"spinlock_wait_cycles":22784,"thread_start_us":37183,"threads_started":1}
I20260928 02:52:56.136123 3518 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42995
I20260928 02:52:56.136447 3518 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:56.136746 3519 heartbeater.cc:347] Connected to a master server at 127.27.229.253:40465
I20260928 02:52:56.136902 3519 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:56.137547 3518 heartbeater.cc:511] Master 127.27.229.254:42995 requested a full tablet report, sending...
I20260928 02:52:56.137549 3519 heartbeater.cc:511] Master 127.27.229.253:40465 requested a full tablet report, sending...
I20260928 02:52:56.138206 3522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.063768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:41245 (local address 127.27.229.252:35049)
0928 02:52:56.088831 (+ 25063us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.088840 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.104332 (+ 15492us) server_negotiation.cc:407] Connection header received
0928 02:52:56.104576 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.104583 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.104664 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.104786 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.105995 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.107110 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.110360 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.111032 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.111121 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.122886 (+ 11765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.122897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.122905 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.122958 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.127613 (+ 4655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.133425 (+ 5812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.133430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.133433 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.133596 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.134925 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.134930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.134934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.135596 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.135731 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.137204 (+ 1473us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.137455 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24856,"thread_start_us":25088,"threads_started":1}
I20260928 02:52:56.139402 2263 ts_manager.cc:194] Re-registered known tserver with Master: 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:56.139540 2863 ts_manager.cc:194] Registered new tserver with Master: 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:56.140066 3523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.064069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.27.229.195:41245)
0928 02:52:56.104213 (+ 40144us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.104238 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.104438 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.104965 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.104970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.105523 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:52:56.105829 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.105840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.108575 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.108580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.110199 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.110213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.112344 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.112350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.112635 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.115729 (+ 3094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:52:56.115756 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:52:56.122644 (+ 6888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:52:56.129157 (+ 6513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.129161 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.129163 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.130735 (+ 1572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.134026 (+ 3291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:52:56.134033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:52:56.134039 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:52:56.134521 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:52:56.135835 (+ 1314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:52:56.135841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:52:56.136669 (+ 828us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.139286 (+ 2617us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.139639 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39940,"mutex_wait_us":582,"thread_start_us":30820,"threads_started":1}
I20260928 02:52:56.140609 2263 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:46853
I20260928 02:52:56.140930 3517 heartbeater.cc:347] Connected to a master server at 127.27.229.252:35049
I20260928 02:52:56.141080 3517 heartbeater.cc:464] Registering TS with master...
I20260928 02:52:56.141520 3517 heartbeater.cc:511] Master 127.27.229.252:35049 requested a full tablet report, sending...
I20260928 02:52:56.142758 3049 ts_manager.cc:194] Registered new tserver with Master: 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:56.144972 3531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.099479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.27.229.254:42995)
0928 02:52:56.104528 (+ 5049us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.104536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.104559 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:52:56.104624 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.104632 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.104724 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.107592 (+ 2868us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.107862 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.109314 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.111905 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.112869 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.113006 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.117778 (+ 4772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.117787 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.117794 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.117845 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.139148 (+ 21303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.140951 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.140956 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.140960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.141130 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.141709 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.141714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.141718 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.143740 (+ 2022us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.144013 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.144335 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.144557 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4893,"thread_start_us":4774,"threads_started":1}
I20260928 02:52:56.150655 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.103210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57756 (local address 127.27.229.253:40465)
0928 02:52:56.120766 (+ 17556us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.120775 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.129231 (+ 8456us) server_negotiation.cc:407] Connection header received
0928 02:52:56.129282 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.129286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.129359 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.129455 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.131801 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.132997 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.135386 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.136679 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.136805 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.142766 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.142773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.142777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.142815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.146762 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.147656 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.147663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.147669 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.147853 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.148338 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.148346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.148350 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.148713 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.148989 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.149986 (+ 997us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.150230 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17353,"thread_start_us":17196,"threads_started":1}
I20260928 02:52:56.153805 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.114270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45628 (local address 127.27.229.252:35049)
0928 02:52:56.122199 (+ 7929us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.122207 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.130935 (+ 8728us) server_negotiation.cc:407] Connection header received
0928 02:52:56.131554 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.131560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.131645 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.131724 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.133968 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.135152 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.141862 (+ 6710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.142557 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.142653 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.146980 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.146987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.146990 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.147021 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.150995 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.151746 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.151751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.151755 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.151891 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.152317 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.152322 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.152326 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.152667 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.152897 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.153184 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.153415 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7789,"thread_start_us":7644,"threads_started":1}
I20260928 02:52:56.149631 2263 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:56.158460 2263 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:56.186901 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.178378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:39513 (local address 127.0.0.1:59852)
0928 02:52:56.179055 (+ 677us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.179080 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.179239 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.180271 (+ 1032us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.180276 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.180302 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:56.180747 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.180764 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.182586 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.182592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.184467 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.184480 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.186053 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.186060 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.186224 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.186263 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.186375 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.186507 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":311,"threads_started":1}
I20260928 02:52:56.186905 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.178607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59852 (local address 127.27.229.194:39513)
0928 02:52:56.179748 (+ 1141us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.179755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.179779 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:52:56.179857 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.179863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.180070 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.180283 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:56.180953 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.182427 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.184663 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.185778 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.185968 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.186168 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.186418 (+ 250us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.186570 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":860,"threads_started":1}
I20260928 02:52:56.192155 3311 tablet_service.cc:1467] Processing CreateTablet for tablet 9743784c4f4e44469929b5d5ec4493ef (DEFAULT_TABLE table=test-table [id=513b9c32cef04f5684b8b3188cf33d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:56.194108 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743784c4f4e44469929b5d5ec4493ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:56.199416 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.180623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41386 (local address 127.27.229.193:41133)
0928 02:52:56.181089 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.181099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.181126 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:52:56.181230 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.181239 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.181471 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.191678 (+ 10207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:56.192311 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.194017 (+ 1706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.196961 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.198132 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.198363 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.198551 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.198906 (+ 355us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.199096 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":588,"threads_started":1}
I20260928 02:52:56.199775 3547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.180301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41133 (local address 127.0.0.1:41386)
0928 02:52:56.180914 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.180932 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.181038 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.191022 (+ 9984us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.191026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.191057 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:56.191439 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.191447 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.194347 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.194353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.196798 (+ 2445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.196811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.198381 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.198485 (+ 104us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.198681 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.198708 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.198823 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.199021 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":1,"spinlock_wait_cycles":7040,"thread_start_us":450,"threads_started":1}
I20260928 02:52:56.201973 3545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.181767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43787 (local address 127.0.0.1:55282)
0928 02:52:56.187174 (+ 5407us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.187192 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.187285 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.193099 (+ 5814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.193107 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.193136 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:56.193316 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.193326 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.196262 (+ 2936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.196267 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.198252 (+ 1985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.198268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.201097 (+ 2829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.201105 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.201296 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.201335 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.201431 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.201629 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5313,"mutex_wait_us":56,"thread_start_us":14196,"threads_started":1}
I20260928 02:52:56.202335 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.182046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55282 (local address 127.27.229.195:43787)
0928 02:52:56.192387 (+ 10341us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.192396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.192435 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:52:56.192540 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.192548 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.192859 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.193503 (+ 644us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:56.193790 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.195707 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.198450 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.200920 (+ 2470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.201120 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.201301 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.201496 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.201695 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10144,"thread_start_us":9932,"threads_started":1}
I20260928 02:52:56.204442 3164 tablet_service.cc:1467] Processing CreateTablet for tablet 9743784c4f4e44469929b5d5ec4493ef (DEFAULT_TABLE table=test-table [id=513b9c32cef04f5684b8b3188cf33d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:56.206010 3164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743784c4f4e44469929b5d5ec4493ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:56.206701 3452 tablet_service.cc:1467] Processing CreateTablet for tablet 9743784c4f4e44469929b5d5ec4493ef (DEFAULT_TABLE table=test-table [id=513b9c32cef04f5684b8b3188cf33d1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:56.208240 3452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9743784c4f4e44469929b5d5ec4493ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:56.211082 3551 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Bootstrap starting.
I20260928 02:52:56.214139 3551 tablet_bootstrap.cc:654] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:56.215405 3551 log.cc:824] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:56.224349 3551 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: No bootstrap required, opened a new log
I20260928 02:52:56.224685 3551 ts_tablet_manager.cc:1397] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260928 02:52:56.225183 3553 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Bootstrap starting.
I20260928 02:52:56.226980 3554 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Bootstrap starting.
I20260928 02:52:56.229067 3553 tablet_bootstrap.cc:654] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:56.230063 3554 tablet_bootstrap.cc:654] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:56.230408 3553 log.cc:824] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:56.231398 3554 log.cc:824] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Log is configured to *not* fsync() on all Append() calls
I20260928 02:52:56.234259 3553 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: No bootstrap required, opened a new log
I20260928 02:52:56.234579 3553 ts_tablet_manager.cc:1397] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260928 02:52:56.234673 3554 tablet_bootstrap.cc:492] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: No bootstrap required, opened a new log
I20260928 02:52:56.234956 3554 ts_tablet_manager.cc:1397] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260928 02:52:56.235589 3551 raft_consensus.cc:348] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.236052 3551 raft_consensus.cc:374] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:56.236151 3551 raft_consensus.cc:729] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6cf40e3b054e839524736121a2bf11, State: Initialized, Role: FOLLOWER
I20260928 02:52:56.236809 3551 consensus_queue.cc:260] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.239193 3377 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
I20260928 02:52:56.245121 3553 raft_consensus.cc:348] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.245474 3551 ts_tablet_manager.cc:1428] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Time spent starting tablet: real 0.021s user 0.011s sys 0.004s
I20260928 02:52:56.245239 3554 raft_consensus.cc:348] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.245555 3553 raft_consensus.cc:374] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:56.245677 3554 raft_consensus.cc:374] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:56.245666 3553 raft_consensus.cc:729] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c3a929c9984276a84754db1d1450a4, State: Initialized, Role: FOLLOWER
I20260928 02:52:56.245779 3554 raft_consensus.cc:729] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695ebf22b5784e60a9830411707b6531, State: Initialized, Role: FOLLOWER
I20260928 02:52:56.246425 3553 consensus_queue.cc:260] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.246447 3554 consensus_queue.cc:260] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.252234 3553 ts_tablet_manager.cc:1428] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Time spent starting tablet: real 0.018s user 0.018s sys 0.001s
I20260928 02:52:56.252368 3554 ts_tablet_manager.cc:1428] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Time spent starting tablet: real 0.017s user 0.015s sys 0.001s
I20260928 02:52:56.252614 3518 heartbeater.cc:503] Master 127.27.229.254:42995 was elected leader, sending a full tablet report...
W20260928 02:52:56.322687 3232 tablet.cc:2369] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:52:56.340283 3520 tablet.cc:2369] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:56.340634 3557 raft_consensus.cc:482] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:56.340927 3557 raft_consensus.cc:504] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.342702 3557 leader_election.cc:288] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133), 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:56.350200 3548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.343126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33173 (local address 127.27.229.193:41133)
0928 02:52:56.343374 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.343383 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.343982 (+ 599us) server_negotiation.cc:407] Connection header received
0928 02:52:56.344248 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.344252 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.344322 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.344425 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:56.344972 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.345798 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.348287 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.349260 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.349445 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.349521 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.349814 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.349933 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:52:56.350361 3560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.343066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41133 (local address 127.27.229.194:33173)
0928 02:52:56.343853 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.343879 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.344043 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.344448 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.344452 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.344478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:56.344833 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.344842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.345934 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.345938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.348106 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.348122 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.349474 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.349481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.349637 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.349669 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.349776 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.349916 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":3,"thread_start_us":424,"threads_started":1}
I20260928 02:52:56.351102 3550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.344108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:47097 (local address 127.27.229.195:43787)
0928 02:52:56.344299 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.344307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.344645 (+ 338us) server_negotiation.cc:407] Connection header received
0928 02:52:56.344864 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.344868 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.344922 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.345011 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:56.345588 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.346395 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.348745 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.349847 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.350104 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.350181 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.350704 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.350862 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 02:52:56.351155 3561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.343887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43787 (local address 127.27.229.194:47097)
0928 02:52:56.344537 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:56.344554 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:56.344713 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:56.345032 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:56.345036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:56.345059 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:56.345425 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.345437 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.346562 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.346567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.348586 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:56.348599 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.350090 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.350097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:56.350307 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.350344 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:52:56.350673 (+ 329us) client_negotiation.cc:240] Negotiation successful
0928 02:52:56.350802 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":441,"threads_started":1}
I20260928 02:52:56.351469 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9743784c4f4e44469929b5d5ec4493ef" candidate_uuid: "2c6cf40e3b054e839524736121a2bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c3a929c9984276a84754db1d1450a4" is_pre_election: true
I20260928 02:52:56.352054 3184 raft_consensus.cc:2454] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6cf40e3b054e839524736121a2bf11 in term 0.
I20260928 02:52:56.352084 3472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9743784c4f4e44469929b5d5ec4493ef" candidate_uuid: "2c6cf40e3b054e839524736121a2bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695ebf22b5784e60a9830411707b6531" is_pre_election: true
I20260928 02:52:56.352602 3472 raft_consensus.cc:2454] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c6cf40e3b054e839524736121a2bf11 in term 0.
I20260928 02:52:56.352794 3264 leader_election.cc:302] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c3a929c9984276a84754db1d1450a4, 2c6cf40e3b054e839524736121a2bf11; no voters:
I20260928 02:52:56.353245 3557 raft_consensus.cc:2781] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:56.353400 3557 raft_consensus.cc:482] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:56.353514 3557 raft_consensus.cc:3037] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:56.355810 3557 raft_consensus.cc:504] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.356462 3557 leader_election.cc:288] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [CANDIDATE]: Term 1 election: Requested vote from peers 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133), 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787)
I20260928 02:52:56.356931 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9743784c4f4e44469929b5d5ec4493ef" candidate_uuid: "2c6cf40e3b054e839524736121a2bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c3a929c9984276a84754db1d1450a4"
I20260928 02:52:56.356971 3472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9743784c4f4e44469929b5d5ec4493ef" candidate_uuid: "2c6cf40e3b054e839524736121a2bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "695ebf22b5784e60a9830411707b6531"
I20260928 02:52:56.357177 3472 raft_consensus.cc:3037] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:56.357169 3184 raft_consensus.cc:3037] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:56.359592 3472 raft_consensus.cc:2454] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6cf40e3b054e839524736121a2bf11 in term 1.
I20260928 02:52:56.359598 3184 raft_consensus.cc:2454] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6cf40e3b054e839524736121a2bf11 in term 1.
I20260928 02:52:56.359984 3264 leader_election.cc:302] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6cf40e3b054e839524736121a2bf11, 695ebf22b5784e60a9830411707b6531; no voters:
I20260928 02:52:56.360268 3557 raft_consensus.cc:2781] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:56.360798 3557 raft_consensus.cc:686] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6cf40e3b054e839524736121a2bf11, State: Running, Role: LEADER
I20260928 02:52:56.361373 3557 consensus_queue.cc:237] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } }
I20260928 02:52:56.366671 2261 catalog_manager.cc:5495] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "2c6cf40e3b054e839524736121a2bf11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:56.398319 3546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.393201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.27.229.194:39513)
0928 02:52:56.393382 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.393387 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.393425 (+ 38us) server_negotiation.cc:407] Connection header received
0928 02:52:56.393510 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.393513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.393587 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.393705 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:56.394044 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.394776 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.396109 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.396513 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.396607 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.397182 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:56.397754 (+ 572us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:56.397871 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.398051 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.398115 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:52:56.406599 3472 raft_consensus.cc:1261] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Refusing update from remote peer 2c6cf40e3b054e839524736121a2bf11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:56.406610 3184 raft_consensus.cc:1261] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Refusing update from remote peer 2c6cf40e3b054e839524736121a2bf11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:56.407624 3562 consensus_queue.cc:1035] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:56.408023 3557 consensus_queue.cc:1035] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:56.423689 3567 mvcc.cc:205] Tried to move back new op lower bound from 7334150047351443456 to 7334150047180107776. Current Snapshot: MvccSnapshot[applied={T|T < 7334150047351443456}]
I20260928 02:52:56.446931 2261 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:56.447237 2261 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:56.459201 2261 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a349fb854e649fcb24b3f62de8a1cc3: Sending DeleteTablet for 3 replicas of tablet 9743784c4f4e44469929b5d5ec4493ef
I20260928 02:52:56.460958 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet 9743784c4f4e44469929b5d5ec4493ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:56 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:59852
I20260928 02:52:56.461581 3452 tablet_service.cc:1514] Processing DeleteTablet for tablet 9743784c4f4e44469929b5d5ec4493ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:56 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:55282
I20260928 02:52:56.463598 3164 tablet_service.cc:1514] Processing DeleteTablet for tablet 9743784c4f4e44469929b5d5ec4493ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:56 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:41386
I20260928 02:52:56.489339 3578 tablet_replica.cc:330] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: stopping tablet replica
I20260928 02:52:56.489852 3578 raft_consensus.cc:2229] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:56.490423 3578 raft_consensus.cc:2258] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:56.491276 3577 tablet_replica.cc:330] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: stopping tablet replica
I20260928 02:52:56.491700 3577 raft_consensus.cc:2229] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:56.492183 3577 raft_consensus.cc:2258] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:56.494168 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2231
I20260928 02:52:56.495476 2293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:56.499248 3578 ts_tablet_manager.cc:1905] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:56.502442 3576 tablet_replica.cc:330] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: stopping tablet replica
I20260928 02:52:56.502888 3576 raft_consensus.cc:2229] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:56.504436 3577 ts_tablet_manager.cc:1905] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:56.511651 3577 ts_tablet_manager.cc:1918] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:56.511911 3577 log.cc:1196] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/9743784c4f4e44469929b5d5ec4493ef
I20260928 02:52:56.512531 3577 ts_tablet_manager.cc:1939] T 9743784c4f4e44469929b5d5ec4493ef P 695ebf22b5784e60a9830411707b6531: Deleting consensus metadata
I20260928 02:52:56.512795 3576 raft_consensus.cc:2258] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:56.516989 3576 ts_tablet_manager.cc:1905] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:56.518865 3578 ts_tablet_manager.cc:1918] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:56.519073 3578 log.cc:1196] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/9743784c4f4e44469929b5d5ec4493ef
I20260928 02:52:56.519680 3578 ts_tablet_manager.cc:1939] T 9743784c4f4e44469929b5d5ec4493ef P 07c3a929c9984276a84754db1d1450a4: Deleting consensus metadata
I20260928 02:52:56.524283 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.468657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57762 (local address 127.27.229.253:40465)
0928 02:52:56.468901 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.468909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.493130 (+ 24221us) server_negotiation.cc:407] Connection header received
0928 02:52:56.493630 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.493637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.493741 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.493846 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.501969 (+ 8123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.503094 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.506357 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.507047 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.507167 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.516877 (+ 9710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.516884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.516889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.516934 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.521190 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.522083 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.522088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.522091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.522238 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.522597 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.522601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.522604 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.523032 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.523274 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.523688 (+ 414us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.523929 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:52:56.525436 3576 ts_tablet_manager.cc:1918] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:56.525627 3576 log.cc:1196] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/9743784c4f4e44469929b5d5ec4493ef
I20260928 02:52:56.526398 3576 ts_tablet_manager.cc:1939] T 9743784c4f4e44469929b5d5ec4493ef P 2c6cf40e3b054e839524736121a2bf11: Deleting consensus metadata
I20260928 02:52:56.541081 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.493479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45636 (local address 127.27.229.252:35049)
0928 02:52:56.493667 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.493673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.524661 (+ 30988us) server_negotiation.cc:407] Connection header received
0928 02:52:56.524923 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.524931 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.525020 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.525091 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.527115 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.528220 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.530077 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.530759 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.530973 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.534442 (+ 3469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.534450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.534456 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.534506 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.538019 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.538970 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.538975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.538980 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.539124 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.539577 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.539582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.539585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.539928 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.540198 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.540509 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.540734 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:52:56.655942 2248 catalog_manager.cc:4841] TS 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513): tablet 9743784c4f4e44469929b5d5ec4493ef (table test-table [id=513b9c32cef04f5684b8b3188cf33d1d]) successfully deleted
I20260928 02:52:56.656553 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2231
W20260928 02:52:56.677691 3117 connection.cc:535] server connection from 127.0.0.1:41386 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:52:56.701704 3534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.682321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57766 (local address 127.27.229.253:40465)
0928 02:52:56.682669 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.682677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.683959 (+ 1282us) server_negotiation.cc:407] Connection header received
0928 02:52:56.684276 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.684284 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.684394 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.684486 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.687355 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.688953 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.690694 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.691279 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.691420 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.694874 (+ 3454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.694882 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.694887 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.694951 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.698494 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.699406 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.699412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.699415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.699573 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.700017 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.700026 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.700032 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.700397 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.700651 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.701043 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.701313 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260928 02:52:56.706135 3535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:56.682321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45642 (local address 127.27.229.252:35049)
0928 02:52:56.682562 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:56.682568 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:56.689236 (+ 6668us) server_negotiation.cc:407] Connection header received
0928 02:52:56.689739 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:56.689745 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:56.689828 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:56.689939 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:56.692212 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.693085 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.694977 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:56.695660 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:56.695803 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:56.699427 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:56.699433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:56.699438 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:56.699489 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:56.703484 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.704200 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.704205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.704209 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.704340 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:56.704739 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:56.704744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:56.704747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:56.705028 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:56.705270 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:56.705600 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:52:56.705811 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260928 02:52:57.430507 3588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.429722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:54119)
0928 02:52:57.430181 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.430311 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":229,"threads_started":1}
W20260928 02:52:57.431030 3518 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:42995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.446507 3589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.445648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:35007)
0928 02:52:57.446210 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.446340 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":340,"threads_started":1}
W20260928 02:52:57.446975 3377 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:42995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.464710 3590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.464028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:38695)
0928 02:52:57.464437 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.464548 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":209,"threads_started":1}
W20260928 02:52:57.465138 3230 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:42995 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.959802 3591 raft_consensus.cc:482] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a349fb854e649fcb24b3f62de8a1cc3)
I20260928 02:52:57.960203 3591 raft_consensus.cc:504] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } }
I20260928 02:52:57.961671 3591 leader_election.cc:288] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995), 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253:40465)
I20260928 02:52:57.963168 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.961910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45894)
0928 02:52:57.962588 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.962878 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":693,"threads_started":1}
W20260928 02:52:57.963434 3035 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.965143 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.964702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45908)
0928 02:52:57.964851 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.964952 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:52:57.965449 3035 leader_election.cc:334] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995): Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.965597 3596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a349fb854e649fcb24b3f62de8a1cc3)
I20260928 02:52:57.965860 3596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } }
I20260928 02:52:57.967334 3596 leader_election.cc:288] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995), a49999b294044db9946303f2140b8f9e (127.27.229.252:35049)
I20260928 02:52:57.968577 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.967404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45914)
0928 02:52:57.967981 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.968166 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":320,"threads_started":1}
W20260928 02:52:57.968977 2849 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.971375 3598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.970680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45926)
0928 02:52:57.970922 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.971098 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260928 02:52:57.971657 2849 leader_election.cc:334] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995): Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.971920 3594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.962252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57770 (local address 127.27.229.253:40465)
0928 02:52:57.963253 (+ 1001us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:57.963260 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:57.963277 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:52:57.963346 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:57.963350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:57.963438 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:57.963551 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:57.964221 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.965082 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.967048 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.968666 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.968880 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:57.968991 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:57.971368 (+ 2377us) server_negotiation.cc:299] Negotiation successful
0928 02:52:57.971519 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":875,"thread_start_us":590,"threads_started":1}
I20260928 02:52:57.971926 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.961910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40465 (local address 127.0.0.1:57770)
0928 02:52:57.962961 (+ 1051us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.962987 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:57.963153 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:57.963598 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:57.963602 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:57.963671 (+ 69us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:57.964066 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:57.964080 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.965257 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.965262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:57.966819 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:57.966892 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.970949 (+ 4057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.970955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:57.971166 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:57.971208 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:52:57.971305 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 02:52:57.971466 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":850,"mutex_wait_us":52,"thread_start_us":263,"threads_started":1}
I20260928 02:52:57.972625 2873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a49999b294044db9946303f2140b8f9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "309a187164ac4f27ae4a5e89274145b2" is_pre_election: true
I20260928 02:52:57.972959 2873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a49999b294044db9946303f2140b8f9e in term 1.
I20260928 02:52:57.973476 3034 leader_election.cc:302] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 309a187164ac4f27ae4a5e89274145b2, a49999b294044db9946303f2140b8f9e; no voters: 5a349fb854e649fcb24b3f62de8a1cc3
I20260928 02:52:57.973868 3591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 02:52:57.973992 3591 raft_consensus.cc:482] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5a349fb854e649fcb24b3f62de8a1cc3)
I20260928 02:52:57.974094 3591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:52:57.976368 3591 raft_consensus.cc:504] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } }
I20260928 02:52:57.976737 3597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.967531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:35049 (local address 127.0.0.1:45644)
0928 02:52:57.970233 (+ 2702us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.970253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:57.970402 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:57.970908 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:57.970913 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:57.970946 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:57.971279 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:57.971295 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.973140 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.973147 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:57.974740 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:57.974752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.975955 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.975962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:57.976148 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:57.976173 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 02:52:57.976277 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:52:57.976422 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2599,"thread_start_us":3126,"threads_started":1}
I20260928 02:52:57.976820 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.968052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45644 (local address 127.27.229.252:35049)
0928 02:52:57.969980 (+ 1928us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:57.969987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:57.970355 (+ 368us) server_negotiation.cc:407] Connection header received
0928 02:52:57.970562 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:57.970571 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:57.970663 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:57.970760 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:57.971564 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.973009 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.974909 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:57.975767 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:57.975955 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:57.976032 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:57.976300 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:52:57.976430 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1767,"thread_start_us":1593,"threads_started":1}
I20260928 02:52:57.977113 3591 leader_election.cc:288] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 election: Requested vote from peers 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995), 309a187164ac4f27ae4a5e89274145b2 (127.27.229.253:40465)
I20260928 02:52:57.977370 3060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "309a187164ac4f27ae4a5e89274145b2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a49999b294044db9946303f2140b8f9e" is_pre_election: true
I20260928 02:52:57.977567 2873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a49999b294044db9946303f2140b8f9e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "309a187164ac4f27ae4a5e89274145b2"
I20260928 02:52:57.977735 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.977377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45934)
0928 02:52:57.977550 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.977620 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 02:52:57.977767 3060 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 309a187164ac4f27ae4a5e89274145b2 in current term 2: Already voted for candidate a49999b294044db9946303f2140b8f9e in this term.
I20260928 02:52:57.977767 2873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:52:57.978343 2850 leader_election.cc:302] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 309a187164ac4f27ae4a5e89274145b2; no voters: 5a349fb854e649fcb24b3f62de8a1cc3, a49999b294044db9946303f2140b8f9e
I20260928 02:52:57.978915 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:57.978433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45936)
0928 02:52:57.978650 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:57.978734 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 02:52:57.979101 3035 leader_election.cc:334] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995): Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
I20260928 02:52:57.980322 2873 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a49999b294044db9946303f2140b8f9e in term 2.
I20260928 02:52:57.980576 3596 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:52:57.980732 3034 leader_election.cc:302] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 309a187164ac4f27ae4a5e89274145b2, a49999b294044db9946303f2140b8f9e; no voters: 5a349fb854e649fcb24b3f62de8a1cc3
I20260928 02:52:57.981004 3591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:52:57.981499 3591 raft_consensus.cc:686] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [term 2 LEADER]: Becoming Leader. State: Replica: a49999b294044db9946303f2140b8f9e, State: Running, Role: LEADER
I20260928 02:52:57.981894 3591 consensus_queue.cc:237] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } }
I20260928 02:52:57.984122 3601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a49999b294044db9946303f2140b8f9e. Latest consensus state: current_term: 2 leader_uuid: "a49999b294044db9946303f2140b8f9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:57.984552 3601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:57.985368 3603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:52:57.989524 3603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:52:57.990432 3603 catalog_manager.cc:1259] Loaded cluster ID: b5f087ef201f423fbebdaba847989d27
I20260928 02:52:57.990501 3603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:52:57.991740 3603 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:52:57.992571 3603 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Loaded TSK: 0
I20260928 02:52:57.993415 3603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:52:58.053748 3049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:52:58.061250 3049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:52:58.065416 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.064979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45952)
0928 02:52:58.065172 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.065259 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:52:58.065502 2873 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [term 2 FOLLOWER]: Refusing update from remote peer a49999b294044db9946303f2140b8f9e: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260928 02:52:58.066581 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.066138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:45960)
0928 02:52:58.066345 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.066439 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:52:58.066491 3601 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260928 02:52:58.066934 3035 consensus_peers.cc:487] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e -> Peer 5a349fb854e649fcb24b3f62de8a1cc3 (127.27.229.254:42995): Couldn't send request to peer 5a349fb854e649fcb24b3f62de8a1cc3. Status: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 02:52:58.069470 3596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a49999b294044db9946303f2140b8f9e. Latest consensus state: current_term: 2 leader_uuid: "a49999b294044db9946303f2140b8f9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:58.069811 3596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:52:58.071375 3601 sys_catalog.cc:455] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a49999b294044db9946303f2140b8f9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:58.071681 3601 sys_catalog.cc:458] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e [sys.catalog]: This master's current role is: LEADER
I20260928 02:52:58.071846 3596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a49999b294044db9946303f2140b8f9e" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "5a349fb854e649fcb24b3f62de8a1cc3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42995 } } peers { permanent_uuid: "309a187164ac4f27ae4a5e89274145b2" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40465 } attrs { promote: false } } peers { permanent_uuid: "a49999b294044db9946303f2140b8f9e" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 35049 } attrs { promote: false } } }
I20260928 02:52:58.072144 3596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 309a187164ac4f27ae4a5e89274145b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:52:58.072247 3605 mvcc.cc:205] Tried to move back new op lower bound from 7334150054149455872 to 7334150053817847808. Current Snapshot: MvccSnapshot[applied={T|T < 7334150054149455872}]
I20260928 02:52:58.091082 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.080729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43787 (local address 127.0.0.1:55286)
0928 02:52:58.080941 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.080962 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:58.081072 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:58.082240 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:58.082248 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:58.082282 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:58.082576 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.082591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.084965 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.084973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.087599 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.087625 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.089938 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.089948 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.090257 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.090322 (+ 65us) client_negotiation.cc:769] Sending connection context
0928 02:52:58.090413 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 02:52:58.090625 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 02:52:58.092703 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.080903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55286 (local address 127.27.229.195:43787)
0928 02:52:58.081816 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.081825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.081849 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:52:58.081935 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.081944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.082051 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.082807 (+ 756us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:58.083072 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.084763 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.087949 (+ 3186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.089686 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.092120 (+ 2434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.092222 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.092277 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.092431 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":768,"thread_start_us":666,"threads_started":1}
I20260928 02:52:58.093447 3452 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c681ee0740a4828896d350295a99 (DEFAULT_TABLE table=test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:58.093891 3452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c681ee0740a4828896d350295a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:58.100046 3613 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Bootstrap starting.
I20260928 02:52:58.101981 3613 tablet_bootstrap.cc:654] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:58.102242 3593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.081248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41133 (local address 127.0.0.1:41392)
0928 02:52:58.081619 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.081634 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:58.081743 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:58.095961 (+ 14218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:58.095966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:58.095993 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:58.096296 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.096310 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.097718 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.097724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.099773 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.099786 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.101452 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.101461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.101681 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.101715 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:52:58.101803 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 02:52:58.101941 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260928 02:52:58.103070 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.081964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.27.229.193:41133)
0928 02:52:58.095538 (+ 13574us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.095547 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.095576 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:52:58.095667 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.095675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.095780 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.095921 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:58.096484 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.097565 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.099958 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.100960 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.101153 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.101261 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.102557 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.102708 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13401,"thread_start_us":14104,"threads_started":1}
I20260928 02:52:58.104020 3164 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c681ee0740a4828896d350295a99 (DEFAULT_TABLE table=test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:58.104161 3613 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: No bootstrap required, opened a new log
I20260928 02:52:58.104362 3613 ts_tablet_manager.cc:1397] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20260928 02:52:58.104451 3164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c681ee0740a4828896d350295a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:58.104977 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.081918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.27.229.194:39513)
0928 02:52:58.098101 (+ 16183us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.098105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.098121 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:52:58.098183 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.098186 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.098240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.098361 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:58.098782 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.099745 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.102725 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.103983 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.104158 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.104229 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.104534 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.104671 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16066,"thread_start_us":15977,"threads_started":1}
I20260928 02:52:58.104983 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.081514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:39513 (local address 127.0.0.1:59874)
0928 02:52:58.091457 (+ 9943us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.091485 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:58.091645 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:58.098390 (+ 6745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:58.098395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:58.098421 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:58.098631 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.098646 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.099908 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.099915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.102563 (+ 2648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.102576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.104136 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.104141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.104360 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.104421 (+ 61us) client_negotiation.cc:769] Sending connection context
0928 02:52:58.104510 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:52:58.104669 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9782,"thread_start_us":10902,"threads_started":1}
I20260928 02:52:58.105151 3613 raft_consensus.cc:348] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.105391 3613 raft_consensus.cc:374] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:58.105759 3311 tablet_service.cc:1467] Processing CreateTablet for tablet 9e07c681ee0740a4828896d350295a99 (DEFAULT_TABLE table=test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:52:58.106164 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e07c681ee0740a4828896d350295a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:52:58.106925 3613 raft_consensus.cc:729] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695ebf22b5784e60a9830411707b6531, State: Initialized, Role: FOLLOWER
I20260928 02:52:58.107203 3613 consensus_queue.cc:260] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.112104 3616 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Bootstrap starting.
I20260928 02:52:58.115113 3616 tablet_bootstrap.cc:654] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:58.115440 3613 ts_tablet_manager.cc:1428] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Time spent starting tablet: real 0.011s user 0.004s sys 0.005s
I20260928 02:52:58.119235 3517 heartbeater.cc:503] Master 127.27.229.252:35049 was elected leader, sending a full tablet report...
I20260928 02:52:58.122146 3617 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Bootstrap starting.
I20260928 02:52:58.123764 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.116061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:47527)
0928 02:52:58.123362 (+ 7301us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.123496 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7146,"thread_start_us":6985,"threads_started":1}
I20260928 02:52:58.124120 3617 tablet_bootstrap.cc:654] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Neither blocks nor log segments found. Creating new log.
I20260928 02:52:58.125074 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.124664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:33737)
0928 02:52:58.124845 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.124942 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:52:58.126641 3617 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: No bootstrap required, opened a new log
I20260928 02:52:58.126799 3617 ts_tablet_manager.cc:1397] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 02:52:58.127628 3617 raft_consensus.cc:348] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.127825 3617 raft_consensus.cc:374] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:58.127915 3617 raft_consensus.cc:729] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6cf40e3b054e839524736121a2bf11, State: Initialized, Role: FOLLOWER
I20260928 02:52:58.128273 3616 tablet_bootstrap.cc:492] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: No bootstrap required, opened a new log
I20260928 02:52:58.128160 3617 consensus_queue.cc:260] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.128438 3616 ts_tablet_manager.cc:1397] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260928 02:52:58.129000 3617 ts_tablet_manager.cc:1428] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 02:52:58.129266 3616 raft_consensus.cc:348] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.129490 3616 raft_consensus.cc:374] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:52:58.129587 3616 raft_consensus.cc:729] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07c3a929c9984276a84754db1d1450a4, State: Initialized, Role: FOLLOWER
I20260928 02:52:58.129863 3616 consensus_queue.cc:260] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.131685 3376 heartbeater.cc:503] Master 127.27.229.252:35049 was elected leader, sending a full tablet report...
I20260928 02:52:58.132858 3616 ts_tablet_manager.cc:1428] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Time spent starting tablet: real 0.004s user 0.002s sys 0.003s
I20260928 02:52:58.135494 3229 heartbeater.cc:503] Master 127.27.229.252:35049 was elected leader, sending a full tablet report...
I20260928 02:52:58.140286 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.133348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:43841)
0928 02:52:58.139276 (+ 5928us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.139955 (+ 679us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5795,"thread_start_us":5293,"threads_started":1}
I20260928 02:52:58.141098 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.140797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:35357)
0928 02:52:58.140946 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.141013 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 02:52:58.142570 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.129553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:46689)
0928 02:52:58.142279 (+ 12726us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.142392 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12614,"threads_started":1}
I20260928 02:52:58.143730 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.143410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:55145)
0928 02:52:58.143554 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.143626 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:52:58.204588 3379 tablet.cc:2369] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:52:58.425763 3615 raft_consensus.cc:482] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:52:58.425990 3615 raft_consensus.cc:504] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.427654 3615 leader_election.cc:288] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133), 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:58.434547 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.428119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:39065 (local address 127.27.229.194:39513)
0928 02:52:58.428332 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.428338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.428364 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:52:58.428499 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.428506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.428609 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.428722 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:58.429245 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.430187 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.432635 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.433757 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.433924 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.434003 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.434196 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.434326 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 02:52:58.434600 3619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.428013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:39513 (local address 127.27.229.195:39065)
0928 02:52:58.428170 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.428190 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:58.428338 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:58.428761 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:58.428772 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:58.428802 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:58.429078 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.429090 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.430333 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.430337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.432464 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.432485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.433930 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.433934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.434044 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.434068 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:52:58.434143 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 02:52:58.434293 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:52:58.435071 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.428177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:36467 (local address 127.27.229.193:41133)
0928 02:52:58.428397 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.428404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.429014 (+ 610us) server_negotiation.cc:407] Connection header received
0928 02:52:58.429275 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.429281 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.429359 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.429462 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:52:58.429895 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.430704 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.432944 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.434140 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.434301 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.434362 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.434666 (+ 304us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.434802 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 02:52:58.435326 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.428034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:41133 (local address 127.27.229.195:36467)
0928 02:52:58.428893 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.428909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:52:58.429029 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:52:58.429496 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:52:58.429500 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:52:58.429519 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:52:58.429754 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.429761 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.430877 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.430883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.432794 (+ 1911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:52:58.432803 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.434327 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.434333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:52:58.434495 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.434524 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:52:58.434627 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:52:58.434752 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"mutex_wait_us":212,"thread_start_us":348,"threads_started":1}
I20260928 02:52:58.435588 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c681ee0740a4828896d350295a99" candidate_uuid: "695ebf22b5784e60a9830411707b6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6cf40e3b054e839524736121a2bf11" is_pre_election: true
I20260928 02:52:58.436005 3331 raft_consensus.cc:2454] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 695ebf22b5784e60a9830411707b6531 in term 0.
I20260928 02:52:58.435999 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c681ee0740a4828896d350295a99" candidate_uuid: "695ebf22b5784e60a9830411707b6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c3a929c9984276a84754db1d1450a4" is_pre_election: true
I20260928 02:52:58.436254 3184 raft_consensus.cc:2454] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 695ebf22b5784e60a9830411707b6531 in term 0.
I20260928 02:52:58.436612 3406 leader_election.cc:302] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c6cf40e3b054e839524736121a2bf11, 695ebf22b5784e60a9830411707b6531; no voters:
I20260928 02:52:58.437044 3615 raft_consensus.cc:2781] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:52:58.437148 3615 raft_consensus.cc:482] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:52:58.437284 3615 raft_consensus.cc:3037] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:58.439471 3615 raft_consensus.cc:504] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.440037 3615 leader_election.cc:288] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [CANDIDATE]: Term 1 election: Requested vote from peers 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133), 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513)
I20260928 02:52:58.440599 3331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c681ee0740a4828896d350295a99" candidate_uuid: "695ebf22b5784e60a9830411707b6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c6cf40e3b054e839524736121a2bf11"
I20260928 02:52:58.440657 3184 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e07c681ee0740a4828896d350295a99" candidate_uuid: "695ebf22b5784e60a9830411707b6531" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07c3a929c9984276a84754db1d1450a4"
I20260928 02:52:58.440828 3331 raft_consensus.cc:3037] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:58.440855 3184 raft_consensus.cc:3037] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:52:58.442924 3184 raft_consensus.cc:2454] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 695ebf22b5784e60a9830411707b6531 in term 1.
I20260928 02:52:58.443737 3405 leader_election.cc:302] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07c3a929c9984276a84754db1d1450a4, 695ebf22b5784e60a9830411707b6531; no voters:
I20260928 02:52:58.443972 3331 raft_consensus.cc:2454] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 695ebf22b5784e60a9830411707b6531 in term 1.
I20260928 02:52:58.444257 3615 raft_consensus.cc:2781] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:52:58.445194 3615 raft_consensus.cc:686] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 1 LEADER]: Becoming Leader. State: Replica: 695ebf22b5784e60a9830411707b6531, State: Running, Role: LEADER
I20260928 02:52:58.445804 3615 consensus_queue.cc:237] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } }
I20260928 02:52:58.446993 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.445842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:53001)
0928 02:52:58.446682 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.446815 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":711}
I20260928 02:52:58.448361 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.447972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:54208)
0928 02:52:58.448152 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.448233 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:52:58.449433 3592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.449035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:54224)
0928 02:52:58.449201 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.449290 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 02:52:58.458099 3049 catalog_manager.cc:5495] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 reported cstate change: term changed from 0 to 1, leader changed from <none> to 695ebf22b5784e60a9830411707b6531 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "695ebf22b5784e60a9830411707b6531" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "695ebf22b5784e60a9830411707b6531" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 } health_report { overall_health: UNKNOWN } } }
I20260928 02:52:58.556306 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.550507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.27.229.195:43787)
0928 02:52:58.550724 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.550733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.551067 (+ 334us) server_negotiation.cc:407] Connection header received
0928 02:52:58.551362 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.551366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.551442 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.551655 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:52:58.552048 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.552756 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.554056 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.554466 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.554623 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.555108 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:52:58.555734 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:52:58.555884 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.555962 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.556042 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:52:58.560797 3184 raft_consensus.cc:1261] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Refusing update from remote peer 695ebf22b5784e60a9830411707b6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:58.560954 3331 raft_consensus.cc:1261] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Refusing update from remote peer 695ebf22b5784e60a9830411707b6531: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:52:58.561929 3615 consensus_queue.cc:1035] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6cf40e3b054e839524736121a2bf11" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 39513 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:58.562269 3632 consensus_queue.cc:1035] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07c3a929c9984276a84754db1d1450a4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:52:58.566453 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.565983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:55021)
0928 02:52:58.566158 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.566255 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 02:52:58.566509 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.565994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:34077)
0928 02:52:58.566230 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.566327 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260928 02:52:58.573879 3634 mvcc.cc:205] Tried to move back new op lower bound from 7334150056177905664 to 7334150055718080512. Current Snapshot: MvccSnapshot[applied={T|T < 7334150056177905664}]
I20260928 02:52:58.574983 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.574512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.194:48181)
0928 02:52:58.574726 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.574817 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:52:58.577553 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.576799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:52799)
0928 02:52:58.577285 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.577390 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386}
I20260928 02:52:58.586161 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.585634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.195:33001)
0928 02:52:58.585870 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.585962 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 02:52:58.587059 3624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.586608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.27.229.193:58383)
0928 02:52:58.586790 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:58.586901 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 02:52:58.595873 3048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:58.596163 3048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:52:58.602025 3048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a49999b294044db9946303f2140b8f9e: Sending DeleteTablet for 3 replicas of tablet 9e07c681ee0740a4828896d350295a99
I20260928 02:52:58.603410 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c681ee0740a4828896d350295a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:58 UTC) from {username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:59874
I20260928 02:52:58.603987 3452 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c681ee0740a4828896d350295a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:58 UTC) from {username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:55286
I20260928 02:52:58.604161 3641 tablet_replica.cc:330] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: stopping tablet replica
I20260928 02:52:58.604372 3641 raft_consensus.cc:2229] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:58.604718 3641 raft_consensus.cc:2258] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:58.604746 3164 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e07c681ee0740a4828896d350295a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:52:58 UTC) from {username='oryx', principal='oryx/127.27.229.252@KRBTEST.COM'} at 127.0.0.1:41392
I20260928 02:52:58.605119 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3053
I20260928 02:52:58.606161 3641 ts_tablet_manager.cc:1905] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:58.615422 3641 ts_tablet_manager.cc:1918] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:58.615625 3641 log.cc:1196] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/9e07c681ee0740a4828896d350295a99
I20260928 02:52:58.616168 3641 ts_tablet_manager.cc:1939] T 9e07c681ee0740a4828896d350295a99 P 2c6cf40e3b054e839524736121a2bf11: Deleting consensus metadata
I20260928 02:52:58.617743 3034 catalog_manager.cc:4841] TS 2c6cf40e3b054e839524736121a2bf11 (127.27.229.194:39513): tablet 9e07c681ee0740a4828896d350295a99 (table test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]) successfully deleted
I20260928 02:52:58.624684 3642 tablet_replica.cc:330] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: stopping tablet replica
I20260928 02:52:58.624995 3642 raft_consensus.cc:2229] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:52:58.625775 3642 raft_consensus.cc:2258] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:58.626585 3643 tablet_replica.cc:330] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: stopping tablet replica
I20260928 02:52:58.626876 3643 raft_consensus.cc:2229] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:52:58.627128 3642 ts_tablet_manager.cc:1905] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:58.627229 3643 raft_consensus.cc:2258] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:52:58.628672 3643 ts_tablet_manager.cc:1905] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:52:58.635321 3642 ts_tablet_manager.cc:1918] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:58.635493 3642 log.cc:1196] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/9e07c681ee0740a4828896d350295a99
I20260928 02:52:58.636081 3642 ts_tablet_manager.cc:1939] T 9e07c681ee0740a4828896d350295a99 P 695ebf22b5784e60a9830411707b6531: Deleting consensus metadata
I20260928 02:52:58.636623 3643 ts_tablet_manager.cc:1918] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:52:58.636827 3643 log.cc:1196] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/9e07c681ee0740a4828896d350295a99
I20260928 02:52:58.637455 3033 catalog_manager.cc:4841] TS 695ebf22b5784e60a9830411707b6531 (127.27.229.195:43787): tablet 9e07c681ee0740a4828896d350295a99 (table test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]) successfully deleted
I20260928 02:52:58.637413 3643 ts_tablet_manager.cc:1939] T 9e07c681ee0740a4828896d350295a99 P 07c3a929c9984276a84754db1d1450a4: Deleting consensus metadata
I20260928 02:52:58.639635 3033 catalog_manager.cc:4841] TS 07c3a929c9984276a84754db1d1450a4 (127.27.229.193:41133): tablet 9e07c681ee0740a4828896d350295a99 (table test-table [id=334a5b7ec3ab4cc0ad4635c28d75605c]) successfully deleted
I20260928 02:52:58.641381 3612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.606899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.27.229.193:41133)
0928 02:52:58.607078 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.607085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.623989 (+ 16904us) server_negotiation.cc:407] Connection header received
0928 02:52:58.624214 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.624220 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.624315 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.624435 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:58.626101 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.627553 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.629454 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.630316 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.630434 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.634211 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:58.634232 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:58.634256 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:58.634297 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:58.638317 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.639337 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.639348 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.639351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.639541 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.640003 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.640007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.640009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.640390 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:58.640631 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.640956 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.641115 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:52:58.642021 3224 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:58.736706 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3053
I20260928 02:52:58.755473 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3241
I20260928 02:52:58.771237 3611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.756339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47498 (local address 127.27.229.194:39513)
0928 02:52:58.756533 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.756540 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.756587 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:52:58.756755 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.756761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.756845 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.756959 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:58.758344 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.759500 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.760792 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.761407 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.761540 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.764368 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:58.764394 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:58.764417 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:58.764467 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:58.768359 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.769095 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.769107 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.769111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.769272 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.769615 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.769620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.769624 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.770076 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:58.770387 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.770627 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.770882 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:52:58.772086 3371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:58.894639 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3241
I20260928 02:52:58.925565 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3389
I20260928 02:52:58.941794 3609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:58.926401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.27.229.195:43787)
0928 02:52:58.926590 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:52:58.926596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:52:58.926616 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:52:58.926805 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:52:58.926811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:52:58.926936 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:52:58.927053 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:52:58.928405 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.929585 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.931084 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:52:58.931742 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:52:58.931898 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:52:58.934418 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:52:58.934442 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:52:58.934463 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:52:58.934511 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:52:58.938629 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.939511 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.939522 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.939526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.939731 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:52:58.940121 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:52:58.940126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:52:58.940129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:52:58.940605 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:52:58.940952 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0928 02:52:58.941168 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 02:52:58.941390 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:52:58.942755 3512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:59.052354 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:59.051586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:54234)
0928 02:52:59.052096 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:59.052194 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":277,"threads_started":1}
I20260928 02:52:59.053656 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:52:59.053289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42995 (local address 127.0.0.1:54244)
0928 02:52:59.053454 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:52:59.053526 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:42995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:52:59.072279 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3389
I20260928 02:52:59.093520 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2726
I20260928 02:52:59.094695 2893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:59.209657 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2726
I20260928 02:52:59.230266 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 2827
I20260928 02:52:59.231482 3080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:52:59.350353 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 2827
2026-09-28T02:52:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (10147 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:52:59 dist-test-slave-s1kg krb5kdc[3654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:52:59 dist-test-slave-s1kg krb5kdc[3654](info): set up 2 sockets
Sep 28 02:52:59 dist-test-slave-s1kg krb5kdc[3654](info): commencing operation
krb5kdc: starting...
W20260928 02:53:00.463987 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:53:00 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:00Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab.
Entry for principal oryx/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab.
I20260928 02:53:00.698364 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41277
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab
--principal=oryx/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:41277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:01.037824 3667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:01.038136 3667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:01.038311 3667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:01.045825 3667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:01.045951 3667 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 02:53:01.046032 3667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:01.046130 3667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:01.046231 3667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:01.046324 3667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:01.056752 3667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:41277
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.254.keytab
--principal=oryx/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:01.058139 3667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:01.059662 3667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:01.072045 3672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:01.072021 3675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:01.072227 3673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:01.073124 3667 server_base.cc:1034] running on GCE node
I20260928 02:53:01.073897 3667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:01.075544 3667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:01.076810 3667 hybrid_clock.cc:648] HybridClock initialized: now 1790563981076750 us; error 41 us; skew 500 ppm
I20260928 02:53:01.077198 3667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:01 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563981, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:01.080675 3667 init.cc:377] Logged in from keytab as oryx/127.27.229.254@KRBTEST.COM (short username oryx)
I20260928 02:53:01.082335 3667 webserver.cc:460] Webserver started at http://127.27.229.254:39291/ using document root <none> and password file <none>
I20260928 02:53:01.083192 3667 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:01.083287 3667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:01.083559 3667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:01.086738 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "b019298cf6254ce2a7dc3e06dfb290ae"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "5918cb4bbfa0c47222d21caa68f49271"
server_key_iv: "ef3fb4671c8dccba60d57d590217f474"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.087642 3667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "b019298cf6254ce2a7dc3e06dfb290ae"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "5918cb4bbfa0c47222d21caa68f49271"
server_key_iv: "ef3fb4671c8dccba60d57d590217f474"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.093397 3667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:53:01.097227 3682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:01.098805 3667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:01.099076 3667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "b019298cf6254ce2a7dc3e06dfb290ae"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "5918cb4bbfa0c47222d21caa68f49271"
server_key_iv: "ef3fb4671c8dccba60d57d590217f474"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.099293 3667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:01.118119 3667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:01.122802 3667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:01.123160 3667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:01.146472 3667 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:41277
I20260928 02:53:01.146508 3743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:41277 every 8 connection(s)
I20260928 02:53:01.147897 3667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:01.152021 3744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:01.152077 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3667
I20260928 02:53:01.152290 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:53:01.152966 28567 external_mini_cluster.cc:1428] Setting key 7332e161958aee5808f8368042deb85b
I20260928 02:53:01.166500 3744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Bootstrap starting.
Sep 28 02:53:01 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:01.170563 3744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:01.172144 3744 log.cc:824] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:01.175674 3744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: No bootstrap required, opened a new log
I20260928 02:53:01.178864 3747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:01.155054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57134 (local address 127.27.229.254:41277)
0928 02:53:01.155910 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:01.155922 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:01.155966 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:53:01.158097 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:01.158124 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:01.158583 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:01.159059 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:01.160893 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.162319 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.164212 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.164939 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.165079 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:01.169855 (+ 4776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:01.169864 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:01.169880 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:01.169926 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:01.173947 (+ 4021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.174805 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.174813 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.174822 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.175075 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.175492 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.175497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.175500 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.176244 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:01.176544 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:01.176950 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 02:53:01.177406 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":629,"thread_start_us":422,"threads_started":1}
I20260928 02:53:01.184940 3744 raft_consensus.cc:348] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } }
I20260928 02:53:01.185420 3744 raft_consensus.cc:374] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:01.185534 3744 raft_consensus.cc:729] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b019298cf6254ce2a7dc3e06dfb290ae, State: Initialized, Role: FOLLOWER
I20260928 02:53:01.186144 3744 consensus_queue.cc:260] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } }
I20260928 02:53:01.186384 3744 raft_consensus.cc:388] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:01.186525 3744 raft_consensus.cc:482] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:01.186697 3744 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:01.188930 3744 raft_consensus.cc:504] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } }
I20260928 02:53:01.189441 3744 leader_election.cc:302] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b019298cf6254ce2a7dc3e06dfb290ae; no voters:
I20260928 02:53:01.190062 3744 leader_election.cc:288] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:01.190284 3749 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:01.191301 3749 raft_consensus.cc:686] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [term 1 LEADER]: Becoming Leader. State: Replica: b019298cf6254ce2a7dc3e06dfb290ae, State: Running, Role: LEADER
I20260928 02:53:01.191905 3749 consensus_queue.cc:237] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } }
I20260928 02:53:01.192406 3744 sys_catalog.cc:564] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:01.194988 3750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } } }
I20260928 02:53:01.194989 3751 sys_catalog.cc:455] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader b019298cf6254ce2a7dc3e06dfb290ae. Latest consensus state: current_term: 1 leader_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b019298cf6254ce2a7dc3e06dfb290ae" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41277 } } }
I20260928 02:53:01.195415 3750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:01.195423 3751 sys_catalog.cc:458] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:01.197880 3755 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:01.203696 3755 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:01.213431 3755 catalog_manager.cc:1347] Generated new cluster ID: 1ffebfdba0454246a02d6a0549efbff7
I20260928 02:53:01.213567 3755 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:01.255441 3755 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:01.256847 3755 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:01.266788 3755 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Generated new TSK 0
I20260928 02:53:01.267777 3755 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
I20260928 02:53:01.335341 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41277
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:01.698422 3770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:01.698724 3770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:01.698907 3770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:01.706548 3770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:01.706663 3770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:01.706804 3770 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:53:01.706986 3770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:01.717195 3770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:01.718503 3770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:01.719930 3770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:01.734126 3776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:01.734169 3778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:01.734098 3775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:01.734858 3770 server_base.cc:1034] running on GCE node
I20260928 02:53:01.735536 3770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:01.736692 3770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:01.737946 3770 hybrid_clock.cc:648] HybridClock initialized: now 1790563981737879 us; error 55 us; skew 500 ppm
I20260928 02:53:01.738265 3770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:01 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563981, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:01.741591 3770 init.cc:377] Logged in from keytab as oryx/127.27.229.193@KRBTEST.COM (short username oryx)
I20260928 02:53:01.743165 3770 webserver.cc:460] Webserver started at http://127.27.229.193:36283/ using document root <none> and password file <none>
I20260928 02:53:01.743884 3770 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:01.744016 3770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:01.744339 3770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:01.747386 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "a5dad8c90c7b8fc3c6a8f74d4fede9e9"
server_key_iv: "56036d6a69cd9e26fc7332ed50bdff3b"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.748214 3770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "a5dad8c90c7b8fc3c6a8f74d4fede9e9"
server_key_iv: "56036d6a69cd9e26fc7332ed50bdff3b"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.753463 3770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:53:01.757109 3785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:01.758797 3770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:01.759064 3770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
format_stamp: "Formatted at 2026-09-28 02:53:01 on dist-test-slave-s1kg"
server_key: "a5dad8c90c7b8fc3c6a8f74d4fede9e9"
server_key_iv: "56036d6a69cd9e26fc7332ed50bdff3b"
server_key_version: "encryptionkey@0"
I20260928 02:53:01.759375 3770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:01.779633 3770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:01.784107 3770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:01.784416 3770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:01.787137 3770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:01.787334 3770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:01.787490 3770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:01.787561 3770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:53:01 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563981, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:01.812666 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:01.788058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:57473 (local address 127.27.229.254:41277)
0928 02:53:01.788507 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:01.788513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:01.789786 (+ 1273us) server_negotiation.cc:407] Connection header received
0928 02:53:01.792059 (+ 2273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:01.792065 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:01.792169 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:01.792306 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:01.795141 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.796413 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.799052 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.799849 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.800003 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:01.805094 (+ 5091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:01.805100 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:01.805103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:01.805137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:01.808503 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.809659 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.809666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.809670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.809832 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.810490 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.810495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.810503 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.810922 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:01.811074 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:01.812067 (+ 993us) server_negotiation.cc:299] Negotiation successful
0928 02:53:01.812295 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":233,"threads_started":1}
I20260928 02:53:01.813755 3795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:01.788780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.193:57473)
0928 02:53:01.789610 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:01.789648 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:01.791765 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:01.792782 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:01.792794 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:01.793834 (+ 1040us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:01.794863 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:01.794890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.796678 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.796685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:01.798833 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:01.798886 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.800170 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.800178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:01.800402 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:01.801154 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:01.801179 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:01.804726 (+ 3547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:01.808683 (+ 3957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:01.808693 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:01.808712 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:01.809473 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:01.809980 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:01.809983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:01.809986 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:01.810361 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:01.811043 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:01.811052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:01.811655 (+ 603us) client_negotiation.cc:769] Sending connection context
0928 02:53:01.811962 (+ 307us) client_negotiation.cc:240] Negotiation successful
0928 02:53:01.812547 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":354,"threads_started":1}
I20260928 02:53:01.835008 3794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:01.820386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:47965 (local address 127.27.229.254:41277)
0928 02:53:01.820645 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:01.820655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:01.821153 (+ 498us) server_negotiation.cc:407] Connection header received
0928 02:53:01.821307 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:01.821316 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:01.821428 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:01.821575 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:01.822731 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.824216 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.825555 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.826341 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.826498 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:01.829108 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:01.829115 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:01.829121 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:01.829167 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:01.832287 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.833024 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.833030 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.833035 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.833157 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:01.833522 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:01.833528 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:01.833532 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:01.833806 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:01.833955 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:01.834372 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 02:53:01.834600 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:53:01.835163 3870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:01.820374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.193:47965)
0928 02:53:01.821030 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:01.821052 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:01.821179 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:01.821631 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:01.821637 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:01.822094 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:01.822556 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:01.822566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.824402 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.824407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:01.825410 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:01.825421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:01.826576 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:01.826584 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:01.826796 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:01.827650 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:01.827669 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:01.828902 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:01.832438 (+ 3536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:01.832444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:01.832448 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:01.832893 (+ 445us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:01.833268 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:01.833274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:01.833278 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:01.833414 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:01.833921 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:01.833926 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:01.834114 (+ 188us) client_negotiation.cc:769] Sending connection context
0928 02:53:01.834277 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:53:01.834620 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":358,"threads_started":1}
I20260928 02:53:01.845149 3770 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:34889
I20260928 02:53:01.845171 3905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:34889 every 8 connection(s)
I20260928 02:53:01.846423 3770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:01.848961 3906 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41277
I20260928 02:53:01.849241 3906 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:01.849923 3906 heartbeater.cc:511] Master 127.27.229.254:41277 requested a full tablet report, sending...
I20260928 02:53:01.852207 3699 ts_manager.cc:194] Registered new tserver with Master: 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889)
I20260928 02:53:01.852840 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3770
I20260928 02:53:01.853024 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:01.853652 28567 external_mini_cluster.cc:1428] Setting key 8ff0f2e32651a5e9ec82dd6765c7c3c3
I20260928 02:53:01.854008 3699 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:57473
WARNING: no policy specified for oryx/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
I20260928 02:53:01.905962 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41277
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:02.245649 3910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:02.246014 3910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:02.246245 3910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:02.254997 3910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:02.255403 3910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:02.255580 3910 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:53:02.255769 3910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:02.267179 3910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:02.268786 3910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:02.270165 3910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:02.283697 3915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:02.283736 3916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:02.284024 3918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:02.284847 3910 server_base.cc:1034] running on GCE node
I20260928 02:53:02.285475 3910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:02.286418 3910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:02.289762 3910 hybrid_clock.cc:648] HybridClock initialized: now 1790563982289677 us; error 44 us; skew 500 ppm
I20260928 02:53:02.290082 3910 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:02.293681 3910 init.cc:377] Logged in from keytab as oryx/127.27.229.194@KRBTEST.COM (short username oryx)
I20260928 02:53:02.295567 3910 webserver.cc:460] Webserver started at http://127.27.229.194:44333/ using document root <none> and password file <none>
I20260928 02:53:02.296293 3910 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:02.296384 3910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:02.296657 3910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:02.300212 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "b7b334564f0334764885e86dc0b8700e"
server_key_iv: "d5a904b10b6698fbff283cebb1538068"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.301046 3910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "b7b334564f0334764885e86dc0b8700e"
server_key_iv: "d5a904b10b6698fbff283cebb1538068"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.307103 3910 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:53:02.310745 3925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:02.312367 3910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:02.312521 3910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "b7b334564f0334764885e86dc0b8700e"
server_key_iv: "d5a904b10b6698fbff283cebb1538068"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.312752 3910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:02.331648 3910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:02.336470 3910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:02.336723 3910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:02.339686 3910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:02.339783 3910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:02.339986 3910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:02.340066 3910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:02.371526 3935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.341471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.194:58669)
0928 02:53:02.342439 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:02.342490 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:02.345151 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:02.346343 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:02.346363 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:02.347376 (+ 1013us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:02.348528 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.348572 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.350282 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.350291 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.353431 (+ 3140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.353464 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.354463 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.354474 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.354943 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.356001 (+ 1058us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:02.356147 (+ 146us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:02.359670 (+ 3523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:02.363442 (+ 3772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.363458 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.363511 (+ 53us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.364791 (+ 1280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.365401 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.365407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.365411 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.365790 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.366915 (+ 1125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:02.366926 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:02.367665 (+ 739us) client_negotiation.cc:769] Sending connection context
0928 02:53:02.368109 (+ 444us) client_negotiation.cc:240] Negotiation successful
0928 02:53:02.369070 (+ 961us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":424,"threads_started":1}
I20260928 02:53:02.372010 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.340853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:58669 (local address 127.27.229.254:41277)
0928 02:53:02.341319 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:02.341324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:02.342652 (+ 1328us) server_negotiation.cc:407] Connection header received
0928 02:53:02.345619 (+ 2967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:02.345628 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:02.345777 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:02.345909 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:02.348778 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.350019 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.353686 (+ 3667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.354152 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.354273 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.360022 (+ 5749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:02.360029 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:02.360032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:02.360070 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:02.363211 (+ 3141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.364983 (+ 1772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.364991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.364995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.365226 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.366102 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.366110 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.366114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.366581 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:02.366784 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:02.371228 (+ 4444us) server_negotiation.cc:299] Negotiation successful
0928 02:53:02.371557 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":254,"threads_started":1}
I20260928 02:53:02.397521 3910 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:40597
I20260928 02:53:02.397529 4045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:40597 every 8 connection(s)
I20260928 02:53:02.399075 3910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:02.399776 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.382520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:36931 (local address 127.27.229.254:41277)
0928 02:53:02.382730 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:02.382736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:02.383617 (+ 881us) server_negotiation.cc:407] Connection header received
0928 02:53:02.383806 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:02.383816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:02.383931 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:02.384075 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:02.385137 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.386598 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.388610 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.389363 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.389522 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.392775 (+ 3253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:02.392785 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:02.392793 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:02.392839 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:02.396278 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.397219 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.397233 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.397237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.397488 (+ 251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.398079 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.398084 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.398087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.398419 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:02.398644 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:02.399091 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 02:53:02.399374 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:53:02.399827 4017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.382384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.194:36931)
0928 02:53:02.383458 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:02.383484 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:02.383660 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:02.384120 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:02.384124 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:02.384583 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:02.384897 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.384907 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.386824 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.386886 (+ 62us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.388414 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.388432 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.389622 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.389630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.389884 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.390714 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:02.390749 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:02.392436 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:02.396515 (+ 4079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.396523 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.396529 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.397059 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.397828 (+ 769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.397832 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.397834 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.397958 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.398547 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:02.398553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:02.398770 (+ 217us) client_negotiation.cc:769] Sending connection context
0928 02:53:02.398974 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 02:53:02.399353 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":697,"threads_started":1}
I20260928 02:53:02.399809 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 3910
I20260928 02:53:02.400007 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:02.400767 28567 external_mini_cluster.cc:1428] Setting key 9d991e7c65291e5c62afc247ea925a24
I20260928 02:53:02.408097 4046 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41277
I20260928 02:53:02.408422 4046 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:02.409217 4046 heartbeater.cc:511] Master 127.27.229.254:41277 requested a full tablet report, sending...
I20260928 02:53:02.410754 3698 ts_manager.cc:194] Registered new tserver with Master: 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597)
I20260928 02:53:02.411707 3698 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:58669
WARNING: no policy specified for oryx/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
I20260928 02:53:02.459514 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41277
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:02.800846 4050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:02.801201 4050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:02.801360 4050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:02.809551 4050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:02.809681 4050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:02.809831 4050 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 02:53:02.809962 4050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:02.820597 4050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.229.254:41277
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:02.821961 4050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:02.823414 4050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:02.838696 4058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:02.838645 4056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:02.838696 4055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:02.839787 4050 server_base.cc:1034] running on GCE node
I20260928 02:53:02.840482 4050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:02.841552 4050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:02.842813 4050 hybrid_clock.cc:648] HybridClock initialized: now 1790563982842742 us; error 55 us; skew 500 ppm
I20260928 02:53:02.843197 4050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:02.846822 4050 init.cc:377] Logged in from keytab as oryx/127.27.229.195@KRBTEST.COM (short username oryx)
I20260928 02:53:02.848681 4050 webserver.cc:460] Webserver started at http://127.27.229.195:46021/ using document root <none> and password file <none>
I20260928 02:53:02.849598 4050 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:02.849707 4050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:02.850054 4050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:02.853314 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "89104a0e9b1849e18bd97607f1619b1a"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "722dd0488dd551fb9ef32e7badf4cdf2"
server_key_iv: "174cd627099dbf751bcb1a8f8e4253b2"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.854076 4050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "89104a0e9b1849e18bd97607f1619b1a"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "722dd0488dd551fb9ef32e7badf4cdf2"
server_key_iv: "174cd627099dbf751bcb1a8f8e4253b2"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.857203 3906 heartbeater.cc:503] Master 127.27.229.254:41277 was elected leader, sending a full tablet report...
I20260928 02:53:02.860061 4050 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 02:53:02.864006 4065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:02.865568 4050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:53:02.865767 4050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "89104a0e9b1849e18bd97607f1619b1a"
format_stamp: "Formatted at 2026-09-28 02:53:02 on dist-test-slave-s1kg"
server_key: "722dd0488dd551fb9ef32e7badf4cdf2"
server_key_iv: "174cd627099dbf751bcb1a8f8e4253b2"
server_key_version: "encryptionkey@0"
I20260928 02:53:02.866010 4050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:02.881477 4050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:02.886180 4050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:02.886569 4050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:02.889407 4050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:02.889533 4050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:02.889684 4050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:02.889763 4050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:02.921305 4074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.891104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.195:51717)
0928 02:53:02.892094 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:02.892145 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:02.895344 (+ 3199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:02.896449 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:02.896469 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:02.897549 (+ 1080us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:02.898801 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.898870 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.900599 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.900604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.903659 (+ 3055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.903679 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.905075 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.905088 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.905527 (+ 439us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.906909 (+ 1382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:02.906951 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:02.911151 (+ 4200us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:02.915703 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.915716 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.915741 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.916576 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.917041 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.917046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.917059 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.917431 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.918125 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:02.918134 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:02.918696 (+ 562us) client_negotiation.cc:769] Sending connection context
0928 02:53:02.919127 (+ 431us) client_negotiation.cc:240] Negotiation successful
0928 02:53:02.919766 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":426,"threads_started":1}
I20260928 02:53:02.922592 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.890352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:51717 (local address 127.27.229.254:41277)
0928 02:53:02.890555 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:02.890563 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:02.892298 (+ 1735us) server_negotiation.cc:407] Connection header received
0928 02:53:02.895797 (+ 3499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:02.895803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:02.895921 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:02.896047 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:02.899120 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.900431 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.903937 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.904763 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.904910 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.911537 (+ 6627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:02.911544 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:02.911547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:02.911585 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:02.915444 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.916742 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.916747 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.916751 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.916900 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.917560 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.917565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.917576 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.917998 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:02.918147 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:02.921963 (+ 3816us) server_negotiation.cc:299] Negotiation successful
0928 02:53:02.922198 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:53:02.930591 4050 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:42333
I20260928 02:53:02.930617 4184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:42333 every 8 connection(s)
I20260928 02:53:02.932066 4050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:02.933830 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4050
I20260928 02:53:02.933991 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:02.934511 28567 external_mini_cluster.cc:1428] Setting key 5807fa62a7ff7bd1b4d9045187dee7d8
I20260928 02:53:02.934525 4185 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41277
I20260928 02:53:02.934849 4185 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:02.936098 4185 heartbeater.cc:511] Master 127.27.229.254:41277 requested a full tablet report, sending...
I20260928 02:53:02.943142 3698 ts_manager.cc:194] Registered new tserver with Master: 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:02.944639 3698 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:51717
I20260928 02:53:02.948273 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.930653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:53077 (local address 127.27.229.254:41277)
0928 02:53:02.930877 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:02.930882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:02.930899 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:53:02.930960 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:02.930964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:02.931020 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:02.931131 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:02.932068 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.932789 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.934583 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.935249 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.935346 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.941669 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:02.941675 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:02.941678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:02.941712 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:02.945452 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.946210 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.946254 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.946257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.946388 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:02.946763 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:02.946768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:02.946771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:02.947105 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:02.947250 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:02.947698 (+ 448us) server_negotiation.cc:299] Negotiation successful
0928 02:53:02.947913 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:53:02.948360 4183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:02.929823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.27.229.195:53077)
0928 02:53:02.930548 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:02.930575 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:02.930728 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:02.931165 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:02.931171 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:02.931634 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:02.931918 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.931927 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.932924 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.932931 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.934398 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:02.934414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.935871 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.935882 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:02.936112 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:02.936927 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:02.936958 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:02.941396 (+ 4438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:02.945606 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.945612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.945617 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.946068 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.946516 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:02.946521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:02.946524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:02.946654 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:02.947232 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:02.947237 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:02.947404 (+ 167us) client_negotiation.cc:769] Sending connection context
0928 02:53:02.947577 (+ 173us) client_negotiation.cc:240] Negotiation successful
0928 02:53:02.947910 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":1085,"threads_started":1}
I20260928 02:53:02.950526 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 02:53:02 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:03 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:53:03.006091 3934 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:02.994136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.27.229.254:41277)
0928 02:53:02.994477 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:02.994482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:02.994706 (+ 224us) server_negotiation.cc:407] Connection header received
0928 02:53:02.994981 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:02.994986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:02.995052 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:02.995138 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:02.996696 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.997580 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.999015 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:02.999533 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:02.999646 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.005846 (+ 6200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57150: BlockingRecv error: recv got EOF from 127.0.0.1:57150 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 28 02:53:03 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563982, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:03.025450 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.009967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57166 (local address 127.27.229.254:41277)
0928 02:53:03.010269 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.010276 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.010670 (+ 394us) server_negotiation.cc:407] Connection header received
0928 02:53:03.010932 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.010937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.011001 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.011088 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:03.012456 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.013261 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.014696 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.015337 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.015447 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.019588 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:03.019594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:03.019598 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:03.019644 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:03.022500 (+ 2856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:03.023433 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:03.023437 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:03.023439 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:03.023570 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:03.024000 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:03.024004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:03.024006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:03.024368 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:03.024648 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.024912 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.025118 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260928 02:53:03.029834 3698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:03.037151 3698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:03.061288 4203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.049561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.0.0.1:47180)
0928 02:53:03.050550 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.050627 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.050879 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.052772 (+ 1893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.052781 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.052819 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.053329 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.053347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.055723 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.055729 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.058408 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.058429 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.060399 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.060404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.060544 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.060594 (+ 50us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.060681 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.060814 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"mutex_wait_us":1,"thread_start_us":464,"threads_started":1}
I20260928 02:53:03.061857 4208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.053015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40597 (local address 127.0.0.1:39264)
0928 02:53:03.053778 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.053792 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.053947 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.054416 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.054421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.054446 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.054963 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.054973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.057087 (+ 2114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.057092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.058982 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.059000 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.060970 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.060978 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.061147 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.061186 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.061290 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.061436 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":19,"spinlock_wait_cycles":1280,"thread_start_us":482,"threads_started":1}
I20260928 02:53:03.062492 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.049855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.27.229.193:34889)
0928 02:53:03.052020 (+ 2165us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.052030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.052062 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:53:03.052177 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.052188 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.052483 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.053565 (+ 1082us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.053792 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.055540 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.058617 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.060179 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.061357 (+ 1178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.061763 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.061839 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.062165 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1998,"thread_start_us":1857,"threads_started":1}
I20260928 02:53:03.062565 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.051886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39264 (local address 127.27.229.194:40597)
0928 02:53:03.053036 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.053045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.053908 (+ 863us) server_negotiation.cc:407] Connection header received
0928 02:53:03.054081 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.054088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.054241 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.054426 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.055137 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.056949 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.059157 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.060739 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.060930 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.061119 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.061992 (+ 873us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.062298 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":990,"thread_start_us":854,"threads_started":1}
I20260928 02:53:03.062688 4204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.049761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:42333 (local address 127.0.0.1:56574)
0928 02:53:03.053236 (+ 3475us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.053263 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.053484 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.054489 (+ 1005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.054494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.054537 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.055029 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.055042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.057050 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.057056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.059405 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.059420 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.061755 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.061762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.061957 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.061989 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.062128 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.062268 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1774,"mutex_wait_us":1,"spinlock_wait_cycles":768,"thread_start_us":1515,"threads_started":1}
I20260928 02:53:03.062754 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.050276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.27.229.195:42333)
0928 02:53:03.052609 (+ 2333us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.052619 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.053365 (+ 746us) server_negotiation.cc:407] Connection header received
0928 02:53:03.053696 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.053705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.053947 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.054148 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.055305 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.056877 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.059624 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.061239 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.061425 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.061587 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.062213 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.062361 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2150,"thread_start_us":1978,"threads_started":1}
I20260928 02:53:03.066253 3976 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4db5980bc4788953c866c531f0e48 (DEFAULT_TABLE table=test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.066256 3836 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4db5980bc4788953c866c531f0e48 (DEFAULT_TABLE table=test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.067600 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4db5980bc4788953c866c531f0e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.067673 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4db5980bc4788953c866c531f0e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.070372 4115 tablet_service.cc:1467] Processing CreateTablet for tablet c1f4db5980bc4788953c866c531f0e48 (DEFAULT_TABLE table=test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.072052 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f4db5980bc4788953c866c531f0e48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.079541 4209 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Bootstrap starting.
I20260928 02:53:03.083105 4209 tablet_bootstrap.cc:654] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.084467 4210 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Bootstrap starting.
I20260928 02:53:03.084647 4209 log.cc:824] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:03.088059 4210 tablet_bootstrap.cc:654] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.088460 4209 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: No bootstrap required, opened a new log
I20260928 02:53:03.088617 4212 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Bootstrap starting.
I20260928 02:53:03.088872 4209 ts_tablet_manager.cc:1397] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260928 02:53:03.089385 4210 log.cc:824] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:03.092483 4212 tablet_bootstrap.cc:654] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.093101 4210 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: No bootstrap required, opened a new log
I20260928 02:53:03.093461 4210 ts_tablet_manager.cc:1397] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260928 02:53:03.093894 4212 log.cc:824] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:03.096977 4212 tablet_bootstrap.cc:492] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: No bootstrap required, opened a new log
I20260928 02:53:03.097280 4212 ts_tablet_manager.cc:1397] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:53:03.100605 4209 raft_consensus.cc:348] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.101171 4209 raft_consensus.cc:374] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.101289 4209 raft_consensus.cc:729] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.102048 4209 consensus_queue.cc:260] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.105846 4212 raft_consensus.cc:348] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.106253 4212 raft_consensus.cc:374] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.106374 4212 raft_consensus.cc:729] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.107209 4046 heartbeater.cc:503] Master 127.27.229.254:41277 was elected leader, sending a full tablet report...
I20260928 02:53:03.106897 4210 raft_consensus.cc:348] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.107206 4212 consensus_queue.cc:260] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.107415 4210 raft_consensus.cc:374] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.107525 4210 raft_consensus.cc:729] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.108184 4209 ts_tablet_manager.cc:1428] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent starting tablet: real 0.019s user 0.014s sys 0.005s
I20260928 02:53:03.108263 4210 consensus_queue.cc:260] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.109829 4185 heartbeater.cc:503] Master 127.27.229.254:41277 was elected leader, sending a full tablet report...
I20260928 02:53:03.110545 4212 ts_tablet_manager.cc:1428] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260928 02:53:03.112474 4210 ts_tablet_manager.cc:1428] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent starting tablet: real 0.019s user 0.010s sys 0.007s
W20260928 02:53:03.166208 4047 tablet.cc:2369] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:53:03.191910 4186 tablet.cc:2369] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:53:03.349119 3907 tablet.cc:2369] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:03.414498 4215 raft_consensus.cc:482] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:03.414880 4215 raft_consensus.cc:504] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.416848 4215 leader_election.cc:288] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.426468 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.417437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33507 (local address 127.27.229.195:42333)
0928 02:53:03.417639 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.417646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.418387 (+ 741us) server_negotiation.cc:407] Connection header received
0928 02:53:03.418630 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.418635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.418737 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.418854 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.419588 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.420733 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.423902 (+ 3169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.425211 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.425371 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.425448 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.426059 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.426227 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:53:03.426450 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.417243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:56669 (local address 127.27.229.193:34889)
0928 02:53:03.417730 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.417739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.418435 (+ 696us) server_negotiation.cc:407] Connection header received
0928 02:53:03.418715 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.418721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.418808 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.418911 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.419688 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.420734 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.423910 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.425195 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.425371 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.425448 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.426059 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.426195 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260928 02:53:03.426705 4218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.417302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.27.229.194:56669)
0928 02:53:03.418247 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.418271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.418497 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.418944 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.418948 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.418967 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.419476 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.419486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.420907 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.420913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.423699 (+ 2786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.423719 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.425554 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.425559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.425812 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.425906 (+ 94us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.426006 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.426188 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"mutex_wait_us":224,"spinlock_wait_cycles":37248,"thread_start_us":535,"threads_started":1}
I20260928 02:53:03.427129 4219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.417358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:42333 (local address 127.27.229.194:33507)
0928 02:53:03.418269 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.418288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.418464 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.418901 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.418909 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.418938 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.419428 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.419442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.420907 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.420913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.423716 (+ 2803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.423725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.425542 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.425550 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.425812 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.425898 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.426006 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.426188 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":791,"mutex_wait_us":234,"thread_start_us":508,"threads_started":1}
I20260928 02:53:03.427836 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4db5980bc4788953c866c531f0e48" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" is_pre_election: true
I20260928 02:53:03.428256 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4db5980bc4788953c866c531f0e48" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a" is_pre_election: true
I20260928 02:53:03.428464 3856 raft_consensus.cc:2454] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 0.
I20260928 02:53:03.428684 4135 raft_consensus.cc:2454] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 0.
I20260928 02:53:03.429224 3927 leader_election.cc:302] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a, 9c41c6f837c740d5af4fdfa4adecb4e8; no voters:
I20260928 02:53:03.429668 4215 raft_consensus.cc:2781] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:03.429840 4215 raft_consensus.cc:482] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:03.429963 4215 raft_consensus.cc:3037] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.432288 4215 raft_consensus.cc:504] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.432974 4215 leader_election.cc:288] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 election: Requested vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.433555 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4db5980bc4788953c866c531f0e48" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
I20260928 02:53:03.433552 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f4db5980bc4788953c866c531f0e48" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a"
I20260928 02:53:03.433772 3856 raft_consensus.cc:3037] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.433799 4135 raft_consensus.cc:3037] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.437068 3856 raft_consensus.cc:2454] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 1.
I20260928 02:53:03.437068 4135 raft_consensus.cc:2454] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 1.
I20260928 02:53:03.437624 3927 leader_election.cc:302] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a, 9c41c6f837c740d5af4fdfa4adecb4e8; no voters:
I20260928 02:53:03.437932 4215 raft_consensus.cc:2781] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:03.438354 4215 raft_consensus.cc:686] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 LEADER]: Becoming Leader. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Running, Role: LEADER
I20260928 02:53:03.438910 4215 consensus_queue.cc:237] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } }
I20260928 02:53:03.444203 3699 catalog_manager.cc:5495] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } health_report { overall_health: HEALTHY } } }
I20260928 02:53:03.522034 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.515983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39270 (local address 127.27.229.194:40597)
0928 02:53:03.516160 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.516181 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.516199 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:53:03.516398 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.516402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.516480 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.516616 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:03.517269 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.518103 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.519670 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.520069 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.520194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.520779 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:03.521365 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:03.521480 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.521630 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.521701 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:03.530778 3856 raft_consensus.cc:1261] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Refusing update from remote peer 9c41c6f837c740d5af4fdfa4adecb4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:03.530779 4135 raft_consensus.cc:1261] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Refusing update from remote peer 9c41c6f837c740d5af4fdfa4adecb4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:03.532004 4220 consensus_queue.cc:1035] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:03.532408 4215 consensus_queue.cc:1035] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:03.544601 4223 mvcc.cc:205] Tried to move back new op lower bound from 7334150076531277824 to 7334150076169592832. Current Snapshot: MvccSnapshot[applied={T|T < 7334150076531277824}]
I20260928 02:53:03.559104 3699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:03.559458 3699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:03.563114 3699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Sending DeleteTablet for 3 replicas of tablet c1f4db5980bc4788953c866c531f0e48
I20260928 02:53:03.564500 4115 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4db5980bc4788953c866c531f0e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:03 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260928 02:53:03.564584 3976 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4db5980bc4788953c866c531f0e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:03 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:39264
I20260928 02:53:03.564577 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f4db5980bc4788953c866c531f0e48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:03 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:47180
I20260928 02:53:03.567318 3699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:03.568264 3699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:03.573899 4233 tablet_replica.cc:330] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: stopping tablet replica
I20260928 02:53:03.574460 4233 raft_consensus.cc:2229] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:03.575049 4233 raft_consensus.cc:2258] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:03.577296 4233 ts_tablet_manager.cc:1905] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:03.580832 3975 tablet_service.cc:1467] Processing CreateTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 (DEFAULT_TABLE table=test-table [id=d2f35ded13534a318da55dcfc88160ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.581189 4115 tablet_service.cc:1467] Processing CreateTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 (DEFAULT_TABLE table=test-table [id=d2f35ded13534a318da55dcfc88160ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.581475 3975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f03f3af8dbc4ae096e4cda85f578a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.581763 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f03f3af8dbc4ae096e4cda85f578a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.587378 3835 tablet_service.cc:1467] Processing CreateTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 (DEFAULT_TABLE table=test-table [id=d2f35ded13534a318da55dcfc88160ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:03.587972 3835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f03f3af8dbc4ae096e4cda85f578a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.588534 4212 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Bootstrap starting.
I20260928 02:53:03.591113 4212 tablet_bootstrap.cc:654] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.593683 4210 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Bootstrap starting.
I20260928 02:53:03.594547 4212 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: No bootstrap required, opened a new log
I20260928 02:53:03.594733 4212 ts_tablet_manager.cc:1397] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260928 02:53:03.595398 4210 tablet_bootstrap.cc:654] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.595672 4209 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Bootstrap starting.
I20260928 02:53:03.595808 4212 raft_consensus.cc:348] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.596107 4212 raft_consensus.cc:374] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.596210 4212 raft_consensus.cc:729] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.596594 4212 consensus_queue.cc:260] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.597697 4209 tablet_bootstrap.cc:654] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.601758 4212 ts_tablet_manager.cc:1428] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20260928 02:53:03.608366 4234 tablet_replica.cc:330] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: stopping tablet replica
I20260928 02:53:03.609048 4234 raft_consensus.cc:2229] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:03.611577 4234 raft_consensus.cc:2258] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:03.614108 4210 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: No bootstrap required, opened a new log
I20260928 02:53:03.614207 4234 ts_tablet_manager.cc:1905] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:03.614428 4210 ts_tablet_manager.cc:1397] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent bootstrapping tablet: real 0.021s user 0.011s sys 0.008s
I20260928 02:53:03.614912 4235 tablet_replica.cc:330] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: stopping tablet replica
I20260928 02:53:03.615442 4235 raft_consensus.cc:2229] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:03.615741 4210 raft_consensus.cc:348] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.616276 4210 raft_consensus.cc:374] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.616559 4210 raft_consensus.cc:729] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.616955 4210 consensus_queue.cc:260] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.619256 4210 ts_tablet_manager.cc:1428] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260928 02:53:03.621009 4233 ts_tablet_manager.cc:1918] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:03.621206 4233 log.cc:1196] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/c1f4db5980bc4788953c866c531f0e48
I20260928 02:53:03.621186 4234 ts_tablet_manager.cc:1918] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:03.621304 4234 log.cc:1196] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/c1f4db5980bc4788953c866c531f0e48
I20260928 02:53:03.621730 4233 ts_tablet_manager.cc:1939] T c1f4db5980bc4788953c866c531f0e48 P 89104a0e9b1849e18bd97607f1619b1a: Deleting consensus metadata
I20260928 02:53:03.621760 4234 ts_tablet_manager.cc:1939] T c1f4db5980bc4788953c866c531f0e48 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting consensus metadata
I20260928 02:53:03.623358 3684 catalog_manager.cc:4841] TS 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889): tablet c1f4db5980bc4788953c866c531f0e48 (table test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]) successfully deleted
I20260928 02:53:03.623004 4235 raft_consensus.cc:2258] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:03.624430 3683 catalog_manager.cc:4841] TS 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333): tablet c1f4db5980bc4788953c866c531f0e48 (table test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]) successfully deleted
I20260928 02:53:03.625028 4209 tablet_bootstrap.cc:492] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: No bootstrap required, opened a new log
I20260928 02:53:03.625217 4209 ts_tablet_manager.cc:1397] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.006s
I20260928 02:53:03.626227 4209 raft_consensus.cc:348] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.626497 4209 raft_consensus.cc:374] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.626612 4209 raft_consensus.cc:729] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.626860 4235 ts_tablet_manager.cc:1905] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:03.626911 4209 consensus_queue.cc:260] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.628468 4209 ts_tablet_manager.cc:1428] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:03.636562 4235 ts_tablet_manager.cc:1918] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:03.636822 4235 log.cc:1196] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/c1f4db5980bc4788953c866c531f0e48
I20260928 02:53:03.637316 4235 ts_tablet_manager.cc:1939] T c1f4db5980bc4788953c866c531f0e48 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting consensus metadata
I20260928 02:53:03.638976 3685 catalog_manager.cc:4841] TS 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597): tablet c1f4db5980bc4788953c866c531f0e48 (table test-table [id=d2b397b6d93a4224987c15ecf6cd75c8]) successfully deleted
I20260928 02:53:03.699065 4217 raft_consensus.cc:482] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:03.699332 4217 raft_consensus.cc:504] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.701022 4217 leader_election.cc:288] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.709419 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.701951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48747 (local address 127.27.229.195:42333)
0928 02:53:03.702128 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.702134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.702998 (+ 864us) server_negotiation.cc:407] Connection header received
0928 02:53:03.703208 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.703213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.703289 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.703353 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.704174 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.705247 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.707208 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.708356 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.708517 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.708601 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.708966 (+ 365us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.709114 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:53:03.709630 4239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.701426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:42333 (local address 127.27.229.193:48747)
0928 02:53:03.702874 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.702895 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.703021 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.703566 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.703571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.703601 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.703990 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.704001 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.705385 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.705390 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.707052 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.707065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.708571 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.708578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.708784 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.708822 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.708923 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.709071 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":4,"thread_start_us":773,"threads_started":1}
I20260928 02:53:03.710400 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f03f3af8dbc4ae096e4cda85f578a40" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a" is_pre_election: true
I20260928 02:53:03.710531 4207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.701495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:46823 (local address 127.27.229.194:40597)
0928 02:53:03.701744 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.701754 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.702527 (+ 773us) server_negotiation.cc:407] Connection header received
0928 02:53:03.702897 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.702902 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.702993 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.703122 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:03.703800 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.705021 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.707727 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.708967 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.709117 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.709181 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.710094 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.710222 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:53:03.710578 4240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.701264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40597 (local address 127.27.229.193:46823)
0928 02:53:03.702387 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.702414 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.702667 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.703152 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.703156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.703182 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:03.703613 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.703623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.705170 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.705178 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.707532 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.707553 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.709341 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.709347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.709875 (+ 528us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.709915 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.710063 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.710201 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":30,"thread_start_us":1274,"threads_started":1}
I20260928 02:53:03.710671 4135 raft_consensus.cc:2454] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 0.
I20260928 02:53:03.711261 3786 leader_election.cc:302] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:03.711416 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f03f3af8dbc4ae096e4cda85f578a40" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" is_pre_election: true
I20260928 02:53:03.711719 4217 raft_consensus.cc:2781] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:03.711851 4217 raft_consensus.cc:482] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:03.711946 4217 raft_consensus.cc:3037] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.711859 3996 raft_consensus.cc:2454] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 0.
I20260928 02:53:03.715178 4217 raft_consensus.cc:504] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.715978 4217 leader_election.cc:288] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 election: Requested vote from peers 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.716568 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f03f3af8dbc4ae096e4cda85f578a40" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a"
I20260928 02:53:03.716794 4135 raft_consensus.cc:3037] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.719103 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f03f3af8dbc4ae096e4cda85f578a40" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
I20260928 02:53:03.719372 3996 raft_consensus.cc:3037] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.719921 4135 raft_consensus.cc:2454] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 1.
I20260928 02:53:03.720346 3786 leader_election.cc:302] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:03.720661 4217 raft_consensus.cc:2781] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:03.721138 4217 raft_consensus.cc:686] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 LEADER]: Becoming Leader. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Running, Role: LEADER
I20260928 02:53:03.721560 4217 consensus_queue.cc:237] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.722693 3996 raft_consensus.cc:2454] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 1.
I20260928 02:53:03.725633 3699 catalog_manager.cc:5495] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a reported cstate change: term changed from 0 to 1, leader changed from <none> to 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193). New cstate: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } health_report { overall_health: UNKNOWN } } }
Sep 28 02:53:03 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563981, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.254@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:03.793877 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.773239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41277 (local address 127.0.0.1:57182)
0928 02:53:03.773983 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.774014 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.774166 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.774707 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.774713 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.775624 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:03.775912 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.775925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.777302 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.777308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.778869 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.778885 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.779738 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.779745 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.779916 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.784161 (+ 4245us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:03.784188 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:03.786979 (+ 2791us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:03.790980 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:03.790998 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:03.791006 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:03.791552 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:03.792027 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:03.792040 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:03.792042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:03.792233 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:03.792896 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:03.792902 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:03.793056 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.793223 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.793440 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":385,"threads_started":1}
I20260928 02:53:03.794023 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.773265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.27.229.254:41277)
0928 02:53:03.774359 (+ 1094us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.774363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.774383 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:53:03.774427 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.774430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.774528 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.774620 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:03.776224 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.777120 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.779057 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.779552 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.779657 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.787246 (+ 7589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:03.787255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:03.787260 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:03.787301 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:03.790775 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:03.791708 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:03.791714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:03.791717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:03.791868 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:03.792354 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:03.792359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:03.792362 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:03.792716 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:03.792866 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.793348 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.793577 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":995,"thread_start_us":817,"threads_started":1}
I20260928 02:53:03.800132 3699 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:57182:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 02:53:03.801433 3699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:03.808914 3836 tablet_service.cc:1467] Processing CreateTablet for tablet 1729bb88f62245ada426b9c1df3708c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615910083784e16aaa933dc0bc30528]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:53:03.809270 4115 tablet_service.cc:1467] Processing CreateTablet for tablet 1729bb88f62245ada426b9c1df3708c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615910083784e16aaa933dc0bc30528]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:53:03.809549 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1729bb88f62245ada426b9c1df3708c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.809413 3976 tablet_service.cc:1467] Processing CreateTablet for tablet 1729bb88f62245ada426b9c1df3708c0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b615910083784e16aaa933dc0bc30528]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 02:53:03.809794 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1729bb88f62245ada426b9c1df3708c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.809952 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1729bb88f62245ada426b9c1df3708c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:03.816962 4212 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: Bootstrap starting.
I20260928 02:53:03.818581 4212 tablet_bootstrap.cc:654] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.818573 4209 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: Bootstrap starting.
I20260928 02:53:03.819082 4210 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: Bootstrap starting.
I20260928 02:53:03.820328 4209 tablet_bootstrap.cc:654] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.820837 4210 tablet_bootstrap.cc:654] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:03.830022 4210 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: No bootstrap required, opened a new log
I20260928 02:53:03.830365 4210 ts_tablet_manager.cc:1397] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.005s
I20260928 02:53:03.831458 4210 raft_consensus.cc:348] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.833029 4210 raft_consensus.cc:374] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.833168 4210 raft_consensus.cc:729] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.833268 4212 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: No bootstrap required, opened a new log
I20260928 02:53:03.833491 4209 tablet_bootstrap.cc:492] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: No bootstrap required, opened a new log
I20260928 02:53:03.833515 4212 ts_tablet_manager.cc:1397] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20260928 02:53:03.833665 4209 ts_tablet_manager.cc:1397] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.002s
I20260928 02:53:03.833508 4210 consensus_queue.cc:260] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.834082 4217 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.834355 4217 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.834424 4212 raft_consensus.cc:348] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.834482 4209 raft_consensus.cc:348] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.834662 4212 raft_consensus.cc:374] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.834714 4209 raft_consensus.cc:374] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:03.834751 4212 raft_consensus.cc:729] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.834791 4209 raft_consensus.cc:729] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Initialized, Role: FOLLOWER
I20260928 02:53:03.835062 4212 consensus_queue.cc:260] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.835091 4209 consensus_queue.cc:260] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.835521 4220 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.835479 4216 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.835800 4216 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.835831 4220 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.836930 4209 ts_tablet_manager.cc:1428] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:03.838171 4210 ts_tablet_manager.cc:1428] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260928 02:53:03.839306 4212 ts_tablet_manager.cc:1428] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:53:03.911906 4217 raft_consensus.cc:482] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:03.912141 4217 raft_consensus.cc:504] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.912766 4217 leader_election.cc:288] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.913307 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1729bb88f62245ada426b9c1df3708c0" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a" is_pre_election: true
I20260928 02:53:03.913337 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1729bb88f62245ada426b9c1df3708c0" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" is_pre_election: true
I20260928 02:53:03.913578 4135 raft_consensus.cc:2454] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 0.
I20260928 02:53:03.913614 3996 raft_consensus.cc:2454] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 0.
I20260928 02:53:03.914062 3786 leader_election.cc:302] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:03.914382 4217 raft_consensus.cc:2781] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:03.914489 4217 raft_consensus.cc:482] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:03.914625 4217 raft_consensus.cc:3037] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.917068 4217 raft_consensus.cc:504] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.917793 4217 leader_election.cc:288] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 election: Requested vote from peers 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:03.918258 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1729bb88f62245ada426b9c1df3708c0" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
I20260928 02:53:03.918370 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1729bb88f62245ada426b9c1df3708c0" candidate_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a"
I20260928 02:53:03.918505 3996 raft_consensus.cc:3037] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.918648 4135 raft_consensus.cc:3037] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:03.920619 4135 raft_consensus.cc:2454] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 1.
I20260928 02:53:03.921042 3786 leader_election.cc:302] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:03.921327 4217 raft_consensus.cc:2781] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:03.921507 4217 raft_consensus.cc:686] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 LEADER]: Becoming Leader. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Running, Role: LEADER
I20260928 02:53:03.921700 3996 raft_consensus.cc:2454] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66bfd28842264513ac4e7f4e6c0ae22a in term 1.
I20260928 02:53:03.921754 4217 consensus_queue.cc:237] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:03.923507 4241 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: TxnStatusTablet state changed. Reason: New leader 66bfd28842264513ac4e7f4e6c0ae22a. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.923951 4241 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:53:03.924686 4254 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 02:53:03.924923 4254 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 02:53:03.925377 3698 catalog_manager.cc:5495] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a reported cstate change: term changed from 0 to 1, leader changed from <none> to 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193). New cstate: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } health_report { overall_health: UNKNOWN } } }
I20260928 02:53:03.969391 4248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.961973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.0.0.1:47188)
0928 02:53:03.962234 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:03.962261 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:03.962421 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:03.963907 (+ 1486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:03.963912 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:03.963928 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:53:03.964175 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.964186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.965468 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.965477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.966594 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:03.966607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.967476 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.967483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:03.967683 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.967744 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:53:03.968740 (+ 996us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:53:03.968763 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:53:03.968869 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:53:03.969033 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:53:03.969762 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.962373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47188 (local address 127.27.229.193:34889)
0928 02:53:03.963490 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.963496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.963521 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:53:03.963614 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.963619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.963709 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.963883 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:03.964351 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.965322 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.966758 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.967302 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.967442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.967993 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:03.968620 (+ 627us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:03.968783 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.968909 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.968995 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":945,"thread_start_us":733,"threads_started":1}
I20260928 02:53:03.974442 4135 raft_consensus.cc:1261] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Refusing update from remote peer 66bfd28842264513ac4e7f4e6c0ae22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:03.975217 3996 raft_consensus.cc:1261] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Refusing update from remote peer 66bfd28842264513ac4e7f4e6c0ae22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:03.975229 4217 consensus_queue.cc:1035] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:03.976100 4217 consensus_queue.cc:1035] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:03.979795 4216 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: TxnStatusTablet state changed. Reason: New leader 66bfd28842264513ac4e7f4e6c0ae22a. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.980094 4216 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.980818 4217 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.981084 4217 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:53:03.983594 4220 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: TxnStatusTablet state changed. Reason: New leader 66bfd28842264513ac4e7f4e6c0ae22a. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.983912 4220 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.984434 4241 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.984691 4241 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 66bfd28842264513ac4e7f4e6c0ae22a: This TxnStatusTablet replica's current role is: LEADER
I20260928 02:53:03.985472 4216 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.985596 4224 mvcc.cc:205] Tried to move back new op lower bound from 7334150078351007744 to 7334150078145863680. Current Snapshot: MvccSnapshot[applied={T|T < 7334150078351007744}]
I20260928 02:53:03.985734 4216 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 89104a0e9b1849e18bd97607f1619b1a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.988277 4220 tablet_replica.cc:439] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } } }
I20260928 02:53:03.988579 4220 tablet_replica.cc:442] T 1729bb88f62245ada426b9c1df3708c0 P 9c41c6f837c740d5af4fdfa4adecb4e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 02:53:03.998669 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:03.993075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.27.229.193:34889)
0928 02:53:03.993273 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:03.993280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:03.993302 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:53:03.993370 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:03.993376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:03.993474 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:03.994122 (+ 648us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:03.994274 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.995252 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.996722 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:03.997223 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:03.997373 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:03.997807 (+ 434us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:03.997913 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:03.998021 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:03.998196 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:53:03.998281 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:53:04.015920 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.010465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39403 (local address 127.27.229.193:34889)
0928 02:53:04.010907 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.010913 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.010969 (+ 56us) server_negotiation.cc:407] Connection header received
0928 02:53:04.011202 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.011215 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.011274 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.011365 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:04.011899 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.012653 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.014088 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.014507 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.014641 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.015119 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.015216 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.015308 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.015408 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.015503 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":298}
I20260928 02:53:04.015975 4262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.010133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.27.229.193:39403)
0928 02:53:04.010885 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:04.010906 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:04.011055 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:04.011375 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:04.011379 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:04.011389 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:53:04.011767 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.011777 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.012775 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.012786 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.013948 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.013960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.014700 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.014707 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.014895 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.014951 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.015308 (+ 357us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.015318 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:53:04.015394 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:53:04.015552 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":30,"thread_start_us":408,"threads_started":1}
I20260928 02:53:04.028362 3996 raft_consensus.cc:1261] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Refusing update from remote peer 66bfd28842264513ac4e7f4e6c0ae22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.029222 4135 raft_consensus.cc:1261] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Refusing update from remote peer 66bfd28842264513ac4e7f4e6c0ae22a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.030493 4217 consensus_queue.cc:1035] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.031349 4217 consensus_queue.cc:1035] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.047511 4225 mvcc.cc:205] Tried to move back new op lower bound from 7334150078571704320 to 7334150077327790080. Current Snapshot: MvccSnapshot[applied={T|T < 7334150078571704320}]
I20260928 02:53:04.131099 3698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.131368 3698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.134325 3698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Sending DeleteTablet for 3 replicas of tablet 9f03f3af8dbc4ae096e4cda85f578a40
I20260928 02:53:04.135260 3976 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:39264
I20260928 02:53:04.135331 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:47180
I20260928 02:53:04.135468 4115 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f03f3af8dbc4ae096e4cda85f578a40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260928 02:53:04.135941 4235 tablet_replica.cc:330] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: stopping tablet replica
I20260928 02:53:04.136128 4270 tablet_replica.cc:330] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: stopping tablet replica
I20260928 02:53:04.136233 4235 raft_consensus.cc:2229] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.136421 4270 raft_consensus.cc:2229] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.136487 4235 raft_consensus.cc:2258] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.136735 4270 raft_consensus.cc:2258] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.137310 4269 tablet_replica.cc:330] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: stopping tablet replica
I20260928 02:53:04.137514 4269 raft_consensus.cc:2229] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:04.137722 4235 ts_tablet_manager.cc:1905] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.138106 4269 raft_consensus.cc:2258] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.139094 4269 ts_tablet_manager.cc:1905] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.140803 4270 ts_tablet_manager.cc:1905] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.147913 4270 ts_tablet_manager.cc:1918] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.148130 4270 log.cc:1196] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/9f03f3af8dbc4ae096e4cda85f578a40
I20260928 02:53:04.148161 4269 ts_tablet_manager.cc:1918] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.148307 4269 log.cc:1196] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/9f03f3af8dbc4ae096e4cda85f578a40
I20260928 02:53:04.148696 4270 ts_tablet_manager.cc:1939] T 9f03f3af8dbc4ae096e4cda85f578a40 P 89104a0e9b1849e18bd97607f1619b1a: Deleting consensus metadata
I20260928 02:53:04.148775 4269 ts_tablet_manager.cc:1939] T 9f03f3af8dbc4ae096e4cda85f578a40 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting consensus metadata
I20260928 02:53:04.149907 3683 catalog_manager.cc:4841] TS 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333): tablet 9f03f3af8dbc4ae096e4cda85f578a40 (table test-table [id=d2f35ded13534a318da55dcfc88160ed]) successfully deleted
I20260928 02:53:04.149976 3684 catalog_manager.cc:4841] TS 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889): tablet 9f03f3af8dbc4ae096e4cda85f578a40 (table test-table [id=d2f35ded13534a318da55dcfc88160ed]) successfully deleted
I20260928 02:53:04.154616 4235 ts_tablet_manager.cc:1918] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.154793 4235 log.cc:1196] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/9f03f3af8dbc4ae096e4cda85f578a40
I20260928 02:53:04.155169 4235 ts_tablet_manager.cc:1939] T 9f03f3af8dbc4ae096e4cda85f578a40 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting consensus metadata
I20260928 02:53:04.156250 3685 catalog_manager.cc:4841] TS 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597): tablet 9f03f3af8dbc4ae096e4cda85f578a40 (table test-table [id=d2f35ded13534a318da55dcfc88160ed]) successfully deleted
Sep 28 02:53:04 dist-test-slave-s1kg krb5kdc[3654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 02:53:04 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:04 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:53:04.208002 4249 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:04.195804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.27.229.254:41277)
0928 02:53:04.196011 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.196018 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.196439 (+ 421us) server_negotiation.cc:407] Connection header received
0928 02:53:04.196609 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.196615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.196699 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.196815 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:04.198170 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.199338 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.200764 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.201421 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.201553 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.207430 (+ 5877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57198: BlockingRecv error: recv got EOF from 127.0.0.1:57198 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 28 02:53:04 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM
I20260928 02:53:04.228979 4249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.210714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57212 (local address 127.27.229.254:41277)
0928 02:53:04.210915 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.210921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.211600 (+ 679us) server_negotiation.cc:407] Connection header received
0928 02:53:04.211762 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.211766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.211851 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.211942 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:04.213487 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.214248 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.216190 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.216930 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.217066 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.222048 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:04.222058 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:04.222063 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:04.222117 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:04.225790 (+ 3673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:04.226931 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:04.226936 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:04.226940 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:04.227110 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:04.227497 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:04.227502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:04.227505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:04.227865 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:04.228121 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.228359 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.228609 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:53:04.232774 3698 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:04.233418 3698 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:04.240269 3836 tablet_service.cc:1467] Processing CreateTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (DEFAULT_TABLE table=test-table [id=d3847229455d4c78a396a8e9cb3270c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.240404 3976 tablet_service.cc:1467] Processing CreateTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (DEFAULT_TABLE table=test-table [id=d3847229455d4c78a396a8e9cb3270c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.240567 4115 tablet_service.cc:1467] Processing CreateTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (DEFAULT_TABLE table=test-table [id=d3847229455d4c78a396a8e9cb3270c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.240772 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d53c36b7084bd2a3d01c90bf5da2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.240832 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d53c36b7084bd2a3d01c90bf5da2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.240969 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4d53c36b7084bd2a3d01c90bf5da2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.247499 4209 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Bootstrap starting.
I20260928 02:53:04.247532 4210 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Bootstrap starting.
I20260928 02:53:04.247646 4212 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Bootstrap starting.
I20260928 02:53:04.249179 4209 tablet_bootstrap.cc:654] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.249188 4212 tablet_bootstrap.cc:654] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.249221 4210 tablet_bootstrap.cc:654] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.259405 4209 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: No bootstrap required, opened a new log
I20260928 02:53:04.259570 4209 ts_tablet_manager.cc:1397] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260928 02:53:04.259954 4212 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: No bootstrap required, opened a new log
I20260928 02:53:04.260303 4209 raft_consensus.cc:348] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.260560 4209 raft_consensus.cc:374] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.260653 4209 raft_consensus.cc:729] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.260497 4212 ts_tablet_manager.cc:1397] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20260928 02:53:04.261835 4209 consensus_queue.cc:260] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.261816 4212 raft_consensus.cc:348] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.262406 4212 raft_consensus.cc:374] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.262567 4212 raft_consensus.cc:729] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.262970 4212 consensus_queue.cc:260] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.266024 4209 ts_tablet_manager.cc:1428] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260928 02:53:04.266750 4212 ts_tablet_manager.cc:1428] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260928 02:53:04.267964 4216 raft_consensus.cc:482] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:04.268170 4216 raft_consensus.cc:504] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.270612 4216 leader_election.cc:288] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597)
I20260928 02:53:04.286038 4210 tablet_bootstrap.cc:492] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: No bootstrap required, opened a new log
I20260928 02:53:04.286238 4210 ts_tablet_manager.cc:1397] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent bootstrapping tablet: real 0.039s user 0.020s sys 0.015s
I20260928 02:53:04.288002 4210 raft_consensus.cc:348] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.288321 4210 raft_consensus.cc:374] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.288416 4210 raft_consensus.cc:729] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.288681 4210 consensus_queue.cc:260] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.289860 4210 ts_tablet_manager.cc:1428] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260928 02:53:04.293979 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.272285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:59287 (local address 127.27.229.193:34889)
0928 02:53:04.272540 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.272549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.286340 (+ 13791us) server_negotiation.cc:407] Connection header received
0928 02:53:04.286573 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.286579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.286663 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.286748 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:04.287673 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.288796 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.291426 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.292852 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.292995 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.293073 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.293575 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.293703 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:53:04.294039 4287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.271665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.27.229.195:59287)
0928 02:53:04.286246 (+ 14581us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:04.286264 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:04.286439 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:04.287037 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:04.287042 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:04.287071 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:04.287465 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.287477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.289110 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.289115 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.291251 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.291266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.293244 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.293250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.293403 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.293429 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:53:04.293536 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:53:04.293686 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14400,"thread_start_us":13598,"threads_started":1}
I20260928 02:53:04.294932 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d53c36b7084bd2a3d01c90bf5da2a8" candidate_uuid: "89104a0e9b1849e18bd97607f1619b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" is_pre_election: true
I20260928 02:53:04.295223 3856 raft_consensus.cc:2454] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89104a0e9b1849e18bd97607f1619b1a in term 0.
I20260928 02:53:04.295341 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.271250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:43811 (local address 127.27.229.194:40597)
0928 02:53:04.277997 (+ 6747us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.278007 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.286419 (+ 8412us) server_negotiation.cc:407] Connection header received
0928 02:53:04.286527 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.286532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.286612 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.286736 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:04.287332 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.288454 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.292701 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.294098 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.294236 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.294307 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.294893 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.295041 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6575,"thread_start_us":6498,"threads_started":1}
I20260928 02:53:04.295842 4067 leader_election.cc:302] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:04.296242 4288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.271211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40597 (local address 127.27.229.195:43811)
0928 02:53:04.285553 (+ 14342us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:04.285575 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:04.285733 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:04.286780 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:04.286787 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:04.286814 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:04.287117 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.287134 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.288660 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.288665 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.292504 (+ 3839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.292518 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.294499 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.294505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.294677 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.294710 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:53:04.295528 (+ 818us) client_negotiation.cc:240] Negotiation successful
0928 02:53:04.295686 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14194,"mutex_wait_us":504,"thread_start_us":14717,"threads_started":1}
I20260928 02:53:04.296416 4216 raft_consensus.cc:2781] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:04.296541 4216 raft_consensus.cc:482] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:04.296643 4216 raft_consensus.cc:3037] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.296939 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d53c36b7084bd2a3d01c90bf5da2a8" candidate_uuid: "89104a0e9b1849e18bd97607f1619b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" is_pre_election: true
I20260928 02:53:04.297184 3996 raft_consensus.cc:2454] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 89104a0e9b1849e18bd97607f1619b1a in term 0.
I20260928 02:53:04.299536 4216 raft_consensus.cc:504] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.300175 4216 leader_election.cc:288] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [CANDIDATE]: Term 1 election: Requested vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597)
I20260928 02:53:04.300735 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d53c36b7084bd2a3d01c90bf5da2a8" candidate_uuid: "89104a0e9b1849e18bd97607f1619b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
I20260928 02:53:04.300976 3856 raft_consensus.cc:3037] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.301622 3996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4d53c36b7084bd2a3d01c90bf5da2a8" candidate_uuid: "89104a0e9b1849e18bd97607f1619b1a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8"
I20260928 02:53:04.301857 3996 raft_consensus.cc:3037] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.303735 3856 raft_consensus.cc:2454] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89104a0e9b1849e18bd97607f1619b1a in term 1.
I20260928 02:53:04.304344 4067 leader_election.cc:302] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 89104a0e9b1849e18bd97607f1619b1a; no voters:
I20260928 02:53:04.304661 4216 raft_consensus.cc:2781] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:04.305063 3996 raft_consensus.cc:2454] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 89104a0e9b1849e18bd97607f1619b1a in term 1.
I20260928 02:53:04.305198 4216 raft_consensus.cc:686] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 1 LEADER]: Becoming Leader. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Running, Role: LEADER
I20260928 02:53:04.305644 4216 consensus_queue.cc:237] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.309381 3699 catalog_manager.cc:5495] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a reported cstate change: term changed from 0 to 1, leader changed from <none> to 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195). New cstate: current_term: 1 leader_uuid: "89104a0e9b1849e18bd97607f1619b1a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } health_report { overall_health: HEALTHY } } }
I20260928 02:53:04.330022 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.320538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.27.229.195:42333)
0928 02:53:04.323432 (+ 2894us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.323438 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.323465 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:53:04.323577 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.323585 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.323726 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.323930 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:04.324522 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.325690 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.327185 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.327852 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.327991 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.328442 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.329225 (+ 783us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.329377 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.329534 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.329641 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2737,"thread_start_us":2620,"threads_started":1}
I20260928 02:53:04.333999 3996 raft_consensus.cc:1261] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Refusing update from remote peer 89104a0e9b1849e18bd97607f1619b1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.334015 3856 raft_consensus.cc:1261] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Refusing update from remote peer 89104a0e9b1849e18bd97607f1619b1a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.334803 4289 consensus_queue.cc:1035] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.335282 4216 consensus_queue.cc:1035] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.365402 3699 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.365705 3699 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.370044 3699 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Sending DeleteTablet for 3 replicas of tablet d4d53c36b7084bd2a3d01c90bf5da2a8
I20260928 02:53:04.371106 3976 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:39264
I20260928 02:53:04.371110 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:47180
I20260928 02:53:04.371402 4269 tablet_replica.cc:330] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: stopping tablet replica
I20260928 02:53:04.371411 4235 tablet_replica.cc:330] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: stopping tablet replica
I20260928 02:53:04.371569 4235 raft_consensus.cc:2229] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.371598 4269 raft_consensus.cc:2229] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.371814 4269 raft_consensus.cc:2258] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.371814 4235 raft_consensus.cc:2258] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.373036 4115 tablet_service.cc:1514] Processing DeleteTablet for tablet d4d53c36b7084bd2a3d01c90bf5da2a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260928 02:53:04.373133 4269 ts_tablet_manager.cc:1905] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.373345 4270 tablet_replica.cc:330] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: stopping tablet replica
I20260928 02:53:04.373576 4270 raft_consensus.cc:2229] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:04.374545 4235 ts_tablet_manager.cc:1905] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.381295 4269 ts_tablet_manager.cc:1918] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:04.381309 4235 ts_tablet_manager.cc:1918] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:04.381475 4269 log.cc:1196] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/d4d53c36b7084bd2a3d01c90bf5da2a8
I20260928 02:53:04.381609 4235 log.cc:1196] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/d4d53c36b7084bd2a3d01c90bf5da2a8
I20260928 02:53:04.382112 4269 ts_tablet_manager.cc:1939] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting consensus metadata
I20260928 02:53:04.382397 4235 ts_tablet_manager.cc:1939] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting consensus metadata
I20260928 02:53:04.383219 3699 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:04.383968 3699 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:04.384338 3685 catalog_manager.cc:4841] TS 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597): tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (table test-table [id=d3847229455d4c78a396a8e9cb3270c8]) successfully deleted
I20260928 02:53:04.387519 3684 catalog_manager.cc:4841] TS 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889): tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (table test-table [id=d3847229455d4c78a396a8e9cb3270c8]) successfully deleted
I20260928 02:53:04.393863 4270 raft_consensus.cc:2258] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.394251 3976 tablet_service.cc:1467] Processing CreateTablet for tablet 76b28efb75144ed69cae21adc753be2f (DEFAULT_TABLE table=test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.394354 4115 tablet_service.cc:1467] Processing CreateTablet for tablet 76b28efb75144ed69cae21adc753be2f (DEFAULT_TABLE table=test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.394730 3976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b28efb75144ed69cae21adc753be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.394739 4115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b28efb75144ed69cae21adc753be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.398028 3836 tablet_service.cc:1467] Processing CreateTablet for tablet 76b28efb75144ed69cae21adc753be2f (DEFAULT_TABLE table=test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:04.398547 3836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b28efb75144ed69cae21adc753be2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:04.401895 4270 ts_tablet_manager.cc:1905] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.407155 4209 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Bootstrap starting.
I20260928 02:53:04.409092 4209 tablet_bootstrap.cc:654] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.411564 4210 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Bootstrap starting.
I20260928 02:53:04.413411 4210 tablet_bootstrap.cc:654] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.414556 4212 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Bootstrap starting.
I20260928 02:53:04.416397 4212 tablet_bootstrap.cc:654] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:04.418797 4270 ts_tablet_manager.cc:1918] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:04.419466 4270 log.cc:1196] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/d4d53c36b7084bd2a3d01c90bf5da2a8
I20260928 02:53:04.420037 4270 ts_tablet_manager.cc:1939] T d4d53c36b7084bd2a3d01c90bf5da2a8 P 89104a0e9b1849e18bd97607f1619b1a: Deleting consensus metadata
I20260928 02:53:04.421321 3683 catalog_manager.cc:4841] TS 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333): tablet d4d53c36b7084bd2a3d01c90bf5da2a8 (table test-table [id=d3847229455d4c78a396a8e9cb3270c8]) successfully deleted
I20260928 02:53:04.423843 4210 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: No bootstrap required, opened a new log
I20260928 02:53:04.424110 4210 ts_tablet_manager.cc:1397] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20260928 02:53:04.425132 4210 raft_consensus.cc:348] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.425514 4210 raft_consensus.cc:374] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.425690 4210 raft_consensus.cc:729] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bfd28842264513ac4e7f4e6c0ae22a, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.426025 4210 consensus_queue.cc:260] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.428380 4210 ts_tablet_manager.cc:1428] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260928 02:53:04.434891 4212 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: No bootstrap required, opened a new log
I20260928 02:53:04.435088 4212 ts_tablet_manager.cc:1397] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.000s
I20260928 02:53:04.435894 4212 raft_consensus.cc:348] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.436070 4209 tablet_bootstrap.cc:492] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: No bootstrap required, opened a new log
I20260928 02:53:04.436164 4212 raft_consensus.cc:374] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.436241 4209 ts_tablet_manager.cc:1397] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.006s
I20260928 02:53:04.436256 4212 raft_consensus.cc:729] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89104a0e9b1849e18bd97607f1619b1a, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.436538 4212 consensus_queue.cc:260] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.437058 4212 ts_tablet_manager.cc:1428] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:53:04.437040 4209 raft_consensus.cc:348] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.437318 4209 raft_consensus.cc:374] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:04.437403 4209 raft_consensus.cc:729] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Initialized, Role: FOLLOWER
I20260928 02:53:04.437654 4209 consensus_queue.cc:260] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.438195 4209 ts_tablet_manager.cc:1428] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260928 02:53:04.595470 4220 raft_consensus.cc:482] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:04.595731 4220 raft_consensus.cc:504] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.596472 4220 leader_election.cc:288] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:04.597193 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b28efb75144ed69cae21adc753be2f" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" is_pre_election: true
I20260928 02:53:04.597255 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b28efb75144ed69cae21adc753be2f" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a" is_pre_election: true
I20260928 02:53:04.597451 3856 raft_consensus.cc:2454] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 0.
I20260928 02:53:04.597514 4135 raft_consensus.cc:2454] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 0.
I20260928 02:53:04.598026 3926 leader_election.cc:302] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 89104a0e9b1849e18bd97607f1619b1a, 9c41c6f837c740d5af4fdfa4adecb4e8; no voters:
I20260928 02:53:04.598381 4220 raft_consensus.cc:2781] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:04.598493 4220 raft_consensus.cc:482] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:04.598593 4220 raft_consensus.cc:3037] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.602092 4220 raft_consensus.cc:504] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.603466 3856 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b28efb75144ed69cae21adc753be2f" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66bfd28842264513ac4e7f4e6c0ae22a"
I20260928 02:53:04.603745 3856 raft_consensus.cc:3037] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.607478 3856 raft_consensus.cc:2454] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 1.
I20260928 02:53:04.608090 3927 leader_election.cc:302] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66bfd28842264513ac4e7f4e6c0ae22a, 9c41c6f837c740d5af4fdfa4adecb4e8; no voters:
I20260928 02:53:04.608971 4135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b28efb75144ed69cae21adc753be2f" candidate_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89104a0e9b1849e18bd97607f1619b1a"
I20260928 02:53:04.609191 4135 raft_consensus.cc:3037] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:04.612185 4135 raft_consensus.cc:2454] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c41c6f837c740d5af4fdfa4adecb4e8 in term 1.
I20260928 02:53:04.614259 4220 leader_election.cc:288] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [CANDIDATE]: Term 1 election: Requested vote from peers 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889), 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333)
I20260928 02:53:04.614533 4220 raft_consensus.cc:2781] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:04.614732 4220 raft_consensus.cc:686] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 LEADER]: Becoming Leader. State: Replica: 9c41c6f837c740d5af4fdfa4adecb4e8, State: Running, Role: LEADER
I20260928 02:53:04.615065 4220 consensus_queue.cc:237] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } }
I20260928 02:53:04.621685 3699 catalog_manager.cc:5495] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c41c6f837c740d5af4fdfa4adecb4e8" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40597 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 } health_report { overall_health: UNKNOWN } } }
I20260928 02:53:04.688652 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.681483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39286 (local address 127.27.229.194:40597)
0928 02:53:04.681669 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.681677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.681696 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:53:04.681786 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.681791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.681879 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.681977 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:04.682622 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.684185 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.686428 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.687128 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.687273 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.687859 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.687996 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.688091 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.688278 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.688365 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:53:04.708729 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.700892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:60763 (local address 127.27.229.193:34889)
0928 02:53:04.701080 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.701086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.701765 (+ 679us) server_negotiation.cc:407] Connection header received
0928 02:53:04.701977 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.701983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.702078 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.702185 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:04.702739 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.704135 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.706169 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.707011 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.707174 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.707745 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.707930 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.708024 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.708273 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.708375 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:53:04.708861 4306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.700914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34889 (local address 127.27.229.194:60763)
0928 02:53:04.701633 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:04.701652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:04.701822 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:04.702220 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:04.702225 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:04.702238 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:53:04.702567 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.702578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.704305 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.704311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.706005 (+ 1694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.706022 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.707205 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.707212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.707446 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.707530 (+ 84us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.708046 (+ 516us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.708059 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 02:53:04.708234 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 02:53:04.708452 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":367,"threads_started":1}
I20260928 02:53:04.720471 4135 raft_consensus.cc:1261] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Refusing update from remote peer 9c41c6f837c740d5af4fdfa4adecb4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.721091 3856 raft_consensus.cc:1261] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Refusing update from remote peer 9c41c6f837c740d5af4fdfa4adecb4e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:04.721746 4300 consensus_queue.cc:1035] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89104a0e9b1849e18bd97607f1619b1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 42333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.722173 4220 consensus_queue.cc:1035] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66bfd28842264513ac4e7f4e6c0ae22a" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:04.774556 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.764502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:51565 (local address 127.27.229.194:40597)
0928 02:53:04.764693 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.764700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.767768 (+ 3068us) server_negotiation.cc:407] Connection header received
0928 02:53:04.767958 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.767964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.768049 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.768160 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:04.768799 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.769971 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.772307 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.772936 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.773036 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.773700 (+ 664us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.773852 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.773913 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.774141 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.774231 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:53:04.775336 4310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.764500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40597 (local address 127.27.229.193:51565)
0928 02:53:04.767638 (+ 3138us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:04.767658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:04.767815 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:04.768204 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:04.768209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:04.768224 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 02:53:04.768606 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.768624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.770236 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.770242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.772130 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:04.772145 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.773243 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.773250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:04.773443 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.773501 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 02:53:04.773992 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 02:53:04.774002 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:53:04.774641 (+ 639us) client_negotiation.cc:240] Negotiation successful
0928 02:53:04.774808 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2972,"thread_start_us":2787,"threads_started":1}
I20260928 02:53:04.826989 3698 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.827287 3698 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57212:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:04.831156 3698 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b019298cf6254ce2a7dc3e06dfb290ae: Sending DeleteTablet for 3 replicas of tablet 76b28efb75144ed69cae21adc753be2f
I20260928 02:53:04.832304 4115 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b28efb75144ed69cae21adc753be2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56574
I20260928 02:53:04.832545 3836 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b28efb75144ed69cae21adc753be2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:47180
I20260928 02:53:04.832651 4270 tablet_replica.cc:330] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: stopping tablet replica
I20260928 02:53:04.832859 4270 raft_consensus.cc:2229] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.832891 4269 tablet_replica.cc:330] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: stopping tablet replica
I20260928 02:53:04.833072 4269 raft_consensus.cc:2229] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:04.833093 4270 raft_consensus.cc:2258] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.833304 4269 raft_consensus.cc:2258] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.834877 4269 ts_tablet_manager.cc:1905] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.834985 4270 ts_tablet_manager.cc:1905] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.839732 3976 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b28efb75144ed69cae21adc753be2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:04 UTC) from {username='oryx', principal='oryx/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:39264
I20260928 02:53:04.840138 4235 tablet_replica.cc:330] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: stopping tablet replica
I20260928 02:53:04.840359 4235 raft_consensus.cc:2229] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:04.840734 4235 raft_consensus.cc:2258] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:04.842285 4235 ts_tablet_manager.cc:1905] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:04.843184 4269 ts_tablet_manager.cc:1918] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.843398 4269 log.cc:1196] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/76b28efb75144ed69cae21adc753be2f
I20260928 02:53:04.843847 4269 ts_tablet_manager.cc:1939] T 76b28efb75144ed69cae21adc753be2f P 66bfd28842264513ac4e7f4e6c0ae22a: Deleting consensus metadata
I20260928 02:53:04.851279 3684 catalog_manager.cc:4841] TS 66bfd28842264513ac4e7f4e6c0ae22a (127.27.229.193:34889): tablet 76b28efb75144ed69cae21adc753be2f (table test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]) successfully deleted
I20260928 02:53:04.853690 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3770
I20260928 02:53:04.859346 4270 ts_tablet_manager.cc:1918] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.850744 4235 ts_tablet_manager.cc:1918] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 02:53:04.859740 4235 log.cc:1196] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/76b28efb75144ed69cae21adc753be2f
I20260928 02:53:04.860318 4235 ts_tablet_manager.cc:1939] T 76b28efb75144ed69cae21adc753be2f P 9c41c6f837c740d5af4fdfa4adecb4e8: Deleting consensus metadata
I20260928 02:53:04.860504 4270 log.cc:1196] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/76b28efb75144ed69cae21adc753be2f
I20260928 02:53:04.861016 4270 ts_tablet_manager.cc:1939] T 76b28efb75144ed69cae21adc753be2f P 89104a0e9b1849e18bd97607f1619b1a: Deleting consensus metadata
I20260928 02:53:04.862454 3685 catalog_manager.cc:4841] TS 9c41c6f837c740d5af4fdfa4adecb4e8 (127.27.229.194:40597): tablet 76b28efb75144ed69cae21adc753be2f (table test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]) successfully deleted
I20260928 02:53:04.864915 3683 catalog_manager.cc:4841] TS 89104a0e9b1849e18bd97607f1619b1a (127.27.229.195:42333): tablet 76b28efb75144ed69cae21adc753be2f (table test-table [id=376d5b5ef95b4b17a0208a09458bd7a0]) successfully deleted
Sep 28 02:53:04 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.193@KRBTEST.COM
I20260928 02:53:04.886651 4257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:04.856095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47198 (local address 127.27.229.193:34889)
0928 02:53:04.856303 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:04.856309 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:04.865838 (+ 9529us) server_negotiation.cc:407] Connection header received
0928 02:53:04.866166 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:04.866173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:04.866291 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:04.866376 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:04.868317 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.869539 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.871710 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:04.872430 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:04.872558 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:04.878255 (+ 5697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:04.878276 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:04.878298 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:04.878346 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:04.882769 (+ 4423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:04.883842 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:04.883852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:04.883856 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:04.884034 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:04.884519 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:04.884524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:04.884527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:04.885036 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:04.885375 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:04.885982 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 02:53:04.886251 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:53:04.888541 3901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:05.079049 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3770
I20260928 02:53:05.104211 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3910
Sep 28 02:53:05 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.194@KRBTEST.COM
I20260928 02:53:05.119535 4286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:05.105019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39290 (local address 127.27.229.194:40597)
0928 02:53:05.105207 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:05.105211 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:05.105229 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:53:05.105388 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:05.105392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:05.105452 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:05.105535 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:05.106738 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:05.107570 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:05.108851 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:05.109304 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:05.109410 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:05.113899 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:05.113915 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:05.113931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:05.113974 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:05.116917 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:05.117631 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:05.117638 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:05.117640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:05.117776 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:05.118118 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:05.118122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:05.118124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:05.118493 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:05.118757 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:05.119025 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 02:53:05.119200 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:53:05.120198 4041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:05.245484 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3910
I20260928 02:53:05.270738 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4050
Sep 28 02:53:05 dist-test-slave-s1kg krb5kdc[3654](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.229.195@KRBTEST.COM
I20260928 02:53:05.287926 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:05.271542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56596 (local address 127.27.229.195:42333)
0928 02:53:05.272117 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:05.272122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:05.272137 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:53:05.272197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:05.272201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:05.272320 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:05.272454 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:05.273857 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:05.274604 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:05.276367 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:05.276807 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:05.276965 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:05.281726 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:05.281743 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:05.281760 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:05.281798 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:05.284860 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:05.285660 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:05.285668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:05.285671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:05.285818 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:05.286248 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:05.286253 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:05.286255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:05.286614 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:05.286924 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:05.287332 (+ 408us) server_negotiation.cc:299] Negotiation successful
0928 02:53:05.287527 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":344,"threads_started":1}
I20260928 02:53:05.288645 4179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:05.421854 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4050
I20260928 02:53:05.443995 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 3667
I20260928 02:53:05.445335 3739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:05.560583 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 3667
2026-09-28T02:53:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6205 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:05 dist-test-slave-s1kg krb5kdc[4321](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:05 dist-test-slave-s1kg krb5kdc[4321](info): set up 2 sockets
Sep 28 02:53:05 dist-test-slave-s1kg krb5kdc[4321](info): commencing operation
krb5kdc: starting...
W20260928 02:53:06.691898 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:53:06 dist-test-slave-s1kg krb5kdc[4321](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:53:06.927033 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46703
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:07.259128 4334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:07.259509 4334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:07.259677 4334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:07.267095 4334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:07.267330 4334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:07.267448 4334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:07.267549 4334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:07.267642 4334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:07.277761 4334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:07.279134 4334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:07.280463 4334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:07.292661 4342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:07.292829 4340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:07.292853 4339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:07.293984 4334 server_base.cc:1034] running on GCE node
I20260928 02:53:07.294902 4334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:07.296494 4334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:07.297730 4334 hybrid_clock.cc:648] HybridClock initialized: now 1790563987297682 us; error 57 us; skew 500 ppm
I20260928 02:53:07.298115 4334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T02:53:07Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1804 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:07 dist-test-slave-s1kg krb5kdc[4348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:07 dist-test-slave-s1kg krb5kdc[4348](info): set up 2 sockets
Sep 28 02:53:07 dist-test-slave-s1kg krb5kdc[4348](info): commencing operation
krb5kdc: starting...
W20260928 02:53:08.511909 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:53:08 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:53:08.743399 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33451
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:33451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:09.075946 4361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:09.076260 4361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:09.076395 4361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:09.084211 4361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:09.084349 4361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:09.084465 4361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:09.084571 4361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:09.084654 4361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:09.094736 4361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:33451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:09.096181 4361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:09.097558 4361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:09.110162 4367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:09.110107 4369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:09.110211 4366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:09.116415 4361 server_base.cc:1034] running on GCE node
I20260928 02:53:09.117329 4361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:09.118964 4361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:09.120234 4361 hybrid_clock.cc:648] HybridClock initialized: now 1790563989120176 us; error 46 us; skew 500 ppm
I20260928 02:53:09.120627 4361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:09 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:09.128067 4361 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:53:09.129849 4361 webserver.cc:460] Webserver started at http://127.27.229.254:40137/ using document root <none> and password file <none>
I20260928 02:53:09.130620 4361 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:09.130782 4361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:09.131135 4361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:09.134220 4361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "de9f33192739495caf2b4bb3cedd2edd"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "59dcdb7608a823d97fe442b48e24ac41"
server_key_iv: "6831d7d9c1e40c08f304e8167349c5c7"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.135185 4361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "de9f33192739495caf2b4bb3cedd2edd"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "59dcdb7608a823d97fe442b48e24ac41"
server_key_iv: "6831d7d9c1e40c08f304e8167349c5c7"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.140637 4361 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:53:09.144264 4376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:09.145839 4361 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:09.146049 4361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "de9f33192739495caf2b4bb3cedd2edd"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "59dcdb7608a823d97fe442b48e24ac41"
server_key_iv: "6831d7d9c1e40c08f304e8167349c5c7"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.146260 4361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:09.193095 4361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:09.197726 4361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:09.198058 4361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:09.217857 4361 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:33451
I20260928 02:53:09.217875 4427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:33451 every 8 connection(s)
I20260928 02:53:09.219274 4361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:09.222975 4428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:09.227344 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4361
I20260928 02:53:09.227562 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:53:09.228193 28567 external_mini_cluster.cc:1428] Setting key 73f6f15c228209f355ce689ea40e866b
I20260928 02:53:09.234205 4428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Bootstrap starting.
I20260928 02:53:09.237608 4428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:09.238754 4428 log.cc:824] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:09.242610 4428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: No bootstrap required, opened a new log
Sep 28 02:53:09 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:09.253002 4428 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:09.253362 4431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:09.230248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59782 (local address 127.27.229.254:33451)
0928 02:53:09.231219 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:09.231231 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:09.231274 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:53:09.233083 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:09.233110 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:09.233539 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:09.233961 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:09.235564 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.236944 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.238310 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.239143 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.239376 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:09.244251 (+ 4875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:09.244262 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:09.244286 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:09.244349 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:09.248661 (+ 4312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:09.249557 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:09.249573 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:09.249582 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:09.249808 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:09.250312 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:09.250321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:09.250330 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:09.251011 (+ 681us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:09.251346 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:09.251821 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 02:53:09.252295 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":383,"threads_started":1}
I20260928 02:53:09.253417 4428 raft_consensus.cc:374] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:09.253537 4428 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9f33192739495caf2b4bb3cedd2edd, State: Initialized, Role: FOLLOWER
I20260928 02:53:09.254287 4428 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:09.254565 4428 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:09.254714 4428 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:09.254957 4428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:09.257520 4428 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:09.258028 4428 leader_election.cc:302] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9f33192739495caf2b4bb3cedd2edd; no voters:
I20260928 02:53:09.258644 4428 leader_election.cc:288] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:09.258853 4433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:09.259552 4433 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 LEADER]: Becoming Leader. State: Replica: de9f33192739495caf2b4bb3cedd2edd, State: Running, Role: LEADER
I20260928 02:53:09.260134 4433 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:09.260872 4428 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:09.263139 4434 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de9f33192739495caf2b4bb3cedd2edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } } }
I20260928 02:53:09.263638 4434 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:09.263948 4435 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9f33192739495caf2b4bb3cedd2edd. Latest consensus state: current_term: 1 leader_uuid: "de9f33192739495caf2b4bb3cedd2edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } } }
I20260928 02:53:09.264299 4435 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:09.265630 4439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:09.271812 4439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:09.280831 4439 catalog_manager.cc:1347] Generated new cluster ID: d5a050eea7d745ddab6d0f1bfe983af6
I20260928 02:53:09.280970 4439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:09.293303 4439 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:09.295001 4439 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:09.303056 4439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Generated new TSK 0
I20260928 02:53:09.304095 4439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:53:09.379520 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:09.730695 4454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:09.731041 4454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:09.731216 4454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:09.738747 4454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:09.738902 4454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:09.739092 4454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:09.749372 4454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:09.750854 4454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:09.752170 4454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:09.766026 4459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:09.765959 4462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:09.765959 4460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:09.767081 4454 server_base.cc:1034] running on GCE node
I20260928 02:53:09.767704 4454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:09.768662 4454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:09.769901 4454 hybrid_clock.cc:648] HybridClock initialized: now 1790563989769855 us; error 41 us; skew 500 ppm
I20260928 02:53:09.770241 4454 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:09 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:09.773577 4454 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:53:09.775058 4454 webserver.cc:460] Webserver started at http://127.27.229.193:37365/ using document root <none> and password file <none>
I20260928 02:53:09.775715 4454 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:09.775815 4454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:09.776050 4454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:09.778977 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "f870fa0003ed4c41bbffdd813ed29b13"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "b43805c568de54e353ba0add5d9b9d93"
server_key_iv: "fbf3a1d3d13080d954cb128febd33128"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.779736 4454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "f870fa0003ed4c41bbffdd813ed29b13"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "b43805c568de54e353ba0add5d9b9d93"
server_key_iv: "fbf3a1d3d13080d954cb128febd33128"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.784936 4454 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:53:09.788399 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:09.789817 4454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:53:09.789963 4454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "f870fa0003ed4c41bbffdd813ed29b13"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "b43805c568de54e353ba0add5d9b9d93"
server_key_iv: "fbf3a1d3d13080d954cb128febd33128"
server_key_version: "encryptionkey@0"
I20260928 02:53:09.790136 4454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:09.815254 4454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:09.819576 4454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:09.819821 4454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:09.820750 4454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:09.822012 4454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:09.822083 4454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:09.822166 4454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:09.822209 4454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:09.854171 4454 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:44193
I20260928 02:53:09.854208 4581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:44193 every 8 connection(s)
I20260928 02:53:09.855805 4454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:09.858422 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4454
I20260928 02:53:09.858618 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:09.859395 28567 external_mini_cluster.cc:1428] Setting key 9e122fef42f47ec9799020f777b1b7b9
Sep 28 02:53:09 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563989, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:09.881793 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:09.858532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:44869 (local address 127.27.229.254:33451)
0928 02:53:09.859189 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:09.859195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:09.860223 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 02:53:09.862988 (+ 2765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:09.862995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:09.863129 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:09.863298 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:09.865574 (+ 2276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.866409 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.869021 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.869469 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.869584 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:09.874408 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:09.874418 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:09.874426 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:09.874476 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:09.878089 (+ 3613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:09.878997 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:09.879006 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:09.879012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:09.879173 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:09.879622 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:09.879630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:09.879637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:09.880023 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:09.880203 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:09.881021 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 02:53:09.881292 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":412,"threads_started":1}
I20260928 02:53:09.882462 4585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:09.859196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:44869)
0928 02:53:09.860046 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:09.860095 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:09.862401 (+ 2306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:09.863620 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:09.863635 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:09.864472 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:09.865319 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:09.865348 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.866582 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.866590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:09.868831 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:09.868856 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:09.869727 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:09.869735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:09.870068 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:09.871171 (+ 1103us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:09.871207 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:09.874140 (+ 2933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:09.878283 (+ 4143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:09.878291 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:09.878308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:09.878780 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:09.879289 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:09.879292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:09.879295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:09.879492 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:09.880221 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:09.880228 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:09.880621 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 02:53:09.880864 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 02:53:09.881265 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":347,"threads_started":1}
I20260928 02:53:09.884878 4582 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33451
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:53:09.885255 4582 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:09.886040 4582 heartbeater.cc:511] Master 127.27.229.254:33451 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:53:09.888589 4393 ts_manager.cc:194] Registered new tserver with Master: f870fa0003ed4c41bbffdd813ed29b13 (127.27.229.193:44193)
I20260928 02:53:09.890038 4393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:44869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:53:09.916460 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:10.248183 4588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:10.248466 4588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:10.248625 4588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:10.256011 4588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:10.256124 4588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:10.256316 4588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:10.266399 4588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:10.267886 4588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:10.269243 4588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:10.282744 4593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:10.282778 4594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:10.282754 4596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:10.283835 4588 server_base.cc:1034] running on GCE node
I20260928 02:53:10.284616 4588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:10.285658 4588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:10.286942 4588 hybrid_clock.cc:648] HybridClock initialized: now 1790563990286897 us; error 34 us; skew 500 ppm
I20260928 02:53:10.287277 4588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:10 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:10.290620 4588 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:53:10.292200 4588 webserver.cc:460] Webserver started at http://127.27.229.194:35795/ using document root <none> and password file <none>
I20260928 02:53:10.292924 4588 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:10.293071 4588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:10.293401 4588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:10.296479 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "5556bf4a99d44a01bf4a6d9a3de6b9fa"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "fcd8f80825b44e9b5c2ec89a5bb52f77"
server_key_iv: "5e7965227c27e42d2d19f509510f9d89"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.297281 4588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "5556bf4a99d44a01bf4a6d9a3de6b9fa"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "fcd8f80825b44e9b5c2ec89a5bb52f77"
server_key_iv: "5e7965227c27e42d2d19f509510f9d89"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.302575 4588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:53:10.306528 4603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.308049 4588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:10.308257 4588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "5556bf4a99d44a01bf4a6d9a3de6b9fa"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "fcd8f80825b44e9b5c2ec89a5bb52f77"
server_key_iv: "5e7965227c27e42d2d19f509510f9d89"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.308471 4588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:10.329006 4588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:10.333530 4588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:10.333840 4588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:10.334807 4588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:10.336169 4588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:10.336273 4588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.336434 4588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:10.336508 4588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.371268 4588 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:38083
I20260928 02:53:10.371328 4715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:38083 every 8 connection(s)
I20260928 02:53:10.372521 4588 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:10.379740 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4588
I20260928 02:53:10.379930 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:10.380474 28567 external_mini_cluster.cc:1428] Setting key d6f2d2220f9e64b17604e2b0719f055d
Sep 28 02:53:10 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:10.394642 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:10.375251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:57043 (local address 127.27.229.254:33451)
0928 02:53:10.375429 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:10.375433 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:10.376616 (+ 1183us) server_negotiation.cc:407] Connection header received
0928 02:53:10.378481 (+ 1865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:10.378487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:10.378588 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:10.378701 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:10.380788 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.381674 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.383599 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.384381 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.384530 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:10.388205 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:10.388214 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:10.388222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:10.388265 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:10.391190 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:10.392214 (+ 1024us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:10.392218 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:10.392221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:10.392352 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:10.392909 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:10.392912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:10.392914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:10.393171 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:10.393277 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:10.394219 (+ 942us) server_negotiation.cc:299] Negotiation successful
0928 02:53:10.394408 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:53:10.396250 4718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:10.375645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:57043)
0928 02:53:10.376444 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:10.376487 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:10.378197 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:10.378962 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:10.378971 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:10.379645 (+ 674us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:10.380504 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:10.380531 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.381830 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.381836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:10.383456 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:10.383468 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.384596 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.384601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:10.384810 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:10.385557 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:10.385584 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:10.387962 (+ 2378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:10.391341 (+ 3379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:10.391353 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:10.391375 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:10.392036 (+ 661us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:10.392470 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:10.392478 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:10.392484 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:10.392780 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:10.393304 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:10.393315 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:10.393781 (+ 466us) client_negotiation.cc:769] Sending connection context
0928 02:53:10.394129 (+ 348us) client_negotiation.cc:240] Negotiation successful
0928 02:53:10.394718 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":319,"threads_started":1}
I20260928 02:53:10.397804 4716 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33451
I20260928 02:53:10.398119 4716 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:10.398923 4716 heartbeater.cc:511] Master 127.27.229.254:33451 requested a full tablet report, sending...
I20260928 02:53:10.400514 4393 ts_manager.cc:194] Registered new tserver with Master: 5556bf4a99d44a01bf4a6d9a3de6b9fa (127.27.229.194:38083)
I20260928 02:53:10.401316 4393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:57043
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:53:10.435647 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:10.771154 4721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:10.771456 4721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:10.771615 4721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:10.779259 4721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:10.779376 4721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:10.779551 4721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:10.789809 4721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:10.791213 4721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:10.792517 4721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:10.805918 4727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:10.806634 4726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:10.806596 4721 server_base.cc:1034] running on GCE node
W20260928 02:53:10.807072 4729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:10.807737 4721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:10.808864 4721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:10.810132 4721 hybrid_clock.cc:648] HybridClock initialized: now 1790563990810081 us; error 43 us; skew 500 ppm
I20260928 02:53:10.810534 4721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:10 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:10.813980 4721 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:53:10.815719 4721 webserver.cc:460] Webserver started at http://127.27.229.195:34543/ using document root <none> and password file <none>
I20260928 02:53:10.816438 4721 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:10.816582 4721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:10.816888 4721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:10.819911 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "13f4b858223e48d7af348c179c059955"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "02b6f986f1b68c3a9809e0c787bff442"
server_key_iv: "3e6101d93e7e5a5e283f9e43eab002c2"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.820708 4721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "13f4b858223e48d7af348c179c059955"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "02b6f986f1b68c3a9809e0c787bff442"
server_key_iv: "3e6101d93e7e5a5e283f9e43eab002c2"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.826054 4721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:53:10.830135 4736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.831770 4721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:10.831983 4721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "13f4b858223e48d7af348c179c059955"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "02b6f986f1b68c3a9809e0c787bff442"
server_key_iv: "3e6101d93e7e5a5e283f9e43eab002c2"
server_key_version: "encryptionkey@0"
I20260928 02:53:10.832204 4721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:10.864257 4721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:10.869345 4721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:10.869767 4721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:10.871058 4721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:10.872506 4721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:10.872614 4721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.872771 4721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:10.872852 4721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:10.893502 4582 heartbeater.cc:503] Master 127.27.229.254:33451 was elected leader, sending a full tablet report...
I20260928 02:53:10.911078 4721 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:38695
I20260928 02:53:10.911120 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:38695 every 8 connection(s)
I20260928 02:53:10.912575 4721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:10.920161 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4721
I20260928 02:53:10.920361 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:10.920889 28567 external_mini_cluster.cc:1428] Setting key 289cd3acdb9ca610b223caedad95de68
Sep 28 02:53:10 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563990, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:10.934762 4851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:10.915331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:53855 (local address 127.27.229.254:33451)
0928 02:53:10.916023 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:10.916031 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:10.918976 (+ 2945us) server_negotiation.cc:407] Connection header received
0928 02:53:10.919045 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:10.919049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:10.919115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:10.919214 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:10.921336 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.922289 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.924499 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.924935 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.925041 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:10.928901 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:10.928906 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:10.928909 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:10.928943 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:10.931654 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:10.932553 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:10.932557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:10.932560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:10.932680 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:10.933149 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:10.933152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:10.933154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:10.933415 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:10.933528 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:10.934263 (+ 735us) server_negotiation.cc:299] Negotiation successful
0928 02:53:10.934444 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260928 02:53:10.936151 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:10.915924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:53855)
0928 02:53:10.916853 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:10.916903 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:10.918603 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:10.919456 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:10.919465 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:10.920204 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:10.921057 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:10.921086 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.922455 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.922461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:10.924339 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:10.924353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:10.925193 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:10.925201 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:10.925434 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:10.926142 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:10.926175 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:10.928656 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:10.931839 (+ 3183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:10.931847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:10.931862 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:10.932427 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:10.932841 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:10.932845 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:10.932848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:10.933040 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:10.933554 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:10.933560 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:10.933916 (+ 356us) client_negotiation.cc:769] Sending connection context
0928 02:53:10.934165 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 02:53:10.934779 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":391,"threads_started":1}
I20260928 02:53:10.937703 4849 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33451
I20260928 02:53:10.938068 4849 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:10.938933 4849 heartbeater.cc:511] Master 127.27.229.254:33451 requested a full tablet report, sending...
I20260928 02:53:10.940600 4393 ts_manager.cc:194] Registered new tserver with Master: 13f4b858223e48d7af348c179c059955 (127.27.229.195:38695)
I20260928 02:53:10.941494 4393 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:53855
I20260928 02:53:10.949839 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
Entry for principal oryx/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab.
WARNING: no policy specified for oryx/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
Entry for principal oryx/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab.
WARNING: no policy specified for oryx/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
Entry for principal oryx/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab.
I20260928 02:53:11.098325 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4454
Sep 28 02:53:11 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:53:11.115365 4860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:11.099488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.27.229.193:44193)
0928 02:53:11.100077 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:11.100084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:11.100111 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:53:11.100353 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:11.100361 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:11.100587 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:11.100744 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:11.102110 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.103282 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.104565 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.105125 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.105258 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:11.109249 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:11.109261 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:11.109276 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:11.109345 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:11.112451 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.113312 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.113319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.113321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.113447 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.113860 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.113863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.113865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.114197 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:11.114457 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:11.114695 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 02:53:11.115065 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":278,"threads_started":1}
I20260928 02:53:11.116653 4577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:11.201653 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4454
I20260928 02:53:11.218964 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4588
Sep 28 02:53:11 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:53:11.236812 4862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:11.219979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47036 (local address 127.27.229.194:38083)
0928 02:53:11.220523 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:11.220530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:11.220555 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:53:11.220610 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:11.220616 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:11.220837 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:11.221016 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:11.222305 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.223835 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.225167 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.225731 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.225873 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:11.230311 (+ 4438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:11.230324 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:11.230374 (+ 50us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:11.230421 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:11.233455 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.234236 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.234244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.234246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.234381 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.235014 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.235020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.235032 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.235428 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:11.235715 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:11.236102 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 02:53:11.236428 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
I20260928 02:53:11.238168 4711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:11.321592 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4588
I20260928 02:53:11.338195 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4721
Sep 28 02:53:11 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:53:11.355382 4864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:11.339220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51694 (local address 127.27.229.195:38695)
0928 02:53:11.339735 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:11.339742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:11.339772 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:53:11.339832 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:11.339839 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:11.340060 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:11.340234 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:11.341614 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.342586 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.344019 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:11.344594 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:11.344747 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:11.349209 (+ 4462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:11.349223 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:11.349240 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:11.349277 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:11.352444 (+ 3167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.353188 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.353196 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.353198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.353323 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:11.353722 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:11.353727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:11.353730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:11.354084 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:11.354411 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:11.354701 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:53:11.355014 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
I20260928 02:53:11.356897 4844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:11.449635 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4721
I20260928 02:53:11.466528 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4361
I20260928 02:53:11.467849 4423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:11.570850 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4361
I20260928 02:53:11.593263 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33451
--webserver_interface=127.27.229.254
--webserver_port=40137
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:33451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:11.931699 4867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:11.932018 4867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:11.932188 4867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:11.939670 4867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:11.939798 4867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:11.939888 4867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:11.939999 4867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:11.940073 4867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:11.950253 4867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:33451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=40137
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:11.951728 4867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:11.953055 4867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:11.965588 4875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:11.965588 4873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:11.965826 4872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:11.966464 4867 server_base.cc:1034] running on GCE node
I20260928 02:53:11.967360 4867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:11.968933 4867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:11.970204 4867 hybrid_clock.cc:648] HybridClock initialized: now 1790563991970156 us; error 84 us; skew 500 ppm
I20260928 02:53:11.970635 4867 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:11 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563991, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:11.974081 4867 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:53:11.975796 4867 webserver.cc:460] Webserver started at http://127.27.229.254:40137/ using document root <none> and password file <none>
I20260928 02:53:11.976540 4867 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:11.976704 4867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:11.982766 4867 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:53:11.986003 4882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:11.987957 4867 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:53:11.988263 4867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "de9f33192739495caf2b4bb3cedd2edd"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "59dcdb7608a823d97fe442b48e24ac41"
server_key_iv: "6831d7d9c1e40c08f304e8167349c5c7"
server_key_version: "encryptionkey@0"
I20260928 02:53:11.989283 4867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:12.019358 4867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:12.024362 4867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:12.024610 4867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:12.044838 4867 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:33451
I20260928 02:53:12.044849 4933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:33451 every 8 connection(s)
I20260928 02:53:12.046483 4867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:12.053756 4934 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:53:12.056874 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4867
I20260928 02:53:12.057202 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Bootstrap starting.
I20260928 02:53:12.057790 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:44193
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:33451
--webserver_port=37365
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:53:12.081889 4934 log.cc:824] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:12.094692 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:53:12.095412 4934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Bootstrap complete.
I20260928 02:53:12.108482 4934 raft_consensus.cc:348] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:12.109551 4934 raft_consensus.cc:729] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de9f33192739495caf2b4bb3cedd2edd, State: Initialized, Role: FOLLOWER
I20260928 02:53:12.110327 4934 consensus_queue.cc:260] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:12.110600 4934 raft_consensus.cc:388] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:12.110735 4934 raft_consensus.cc:482] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:12.110980 4934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:53:12.115687 4934 raft_consensus.cc:504] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:12.116263 4934 leader_election.cc:302] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de9f33192739495caf2b4bb3cedd2edd; no voters:
I20260928 02:53:12.116910 4934 leader_election.cc:288] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:53:12.117131 4938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:53:12.118914 4934 sys_catalog.cc:564] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:12.124168 4938 raft_consensus.cc:686] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [term 2 LEADER]: Becoming Leader. State: Replica: de9f33192739495caf2b4bb3cedd2edd, State: Running, Role: LEADER
I20260928 02:53:12.125036 4938 consensus_queue.cc:237] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } }
I20260928 02:53:12.128620 4939 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de9f33192739495caf2b4bb3cedd2edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } } }
I20260928 02:53:12.129078 4939 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:12.131134 4940 sys_catalog.cc:455] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: SysCatalogTable state changed. Reason: New leader de9f33192739495caf2b4bb3cedd2edd. Latest consensus state: current_term: 2 leader_uuid: "de9f33192739495caf2b4bb3cedd2edd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de9f33192739495caf2b4bb3cedd2edd" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33451 } } }
I20260928 02:53:12.132421 4940 sys_catalog.cc:458] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:12.144438 4952 catalog_manager.cc:1259] Loaded cluster ID: d5a050eea7d745ddab6d0f1bfe983af6
I20260928 02:53:12.144584 4952 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: loading cluster ID for follower catalog manager: success
I20260928 02:53:12.147456 4952 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: acquiring CA information for follower catalog manager: success
I20260928 02:53:12.149284 4952 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:53:12.149507 4953 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:12.152027 4953 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:12.153190 4953 catalog_manager.cc:1259] Loaded cluster ID: d5a050eea7d745ddab6d0f1bfe983af6
I20260928 02:53:12.153299 4953 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:12.155231 4953 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:12.156369 4953 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de9f33192739495caf2b4bb3cedd2edd: Loaded TSK: 0
I20260928 02:53:12.157395 4953 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:53:12.429499 4936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:12.429795 4936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:12.429947 4936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:12.437429 4936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:12.437544 4936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:12.437721 4936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:12.447733 4936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.193.keytab
--principal=oryx/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:44193
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=37365
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:12.449126 4936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:12.450440 4936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:12.463954 4960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:12.463954 4961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:12.464746 4963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:12.465211 4936 server_base.cc:1034] running on GCE node
I20260928 02:53:12.465909 4936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:12.467108 4936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:12.468339 4936 hybrid_clock.cc:648] HybridClock initialized: now 1790563992468278 us; error 46 us; skew 500 ppm
I20260928 02:53:12.468701 4936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:12.472698 4936 init.cc:377] Logged in from keytab as oryx/127.27.229.193@KRBTEST.COM (short username oryx)
I20260928 02:53:12.474493 4936 webserver.cc:460] Webserver started at http://127.27.229.193:37365/ using document root <none> and password file <none>
I20260928 02:53:12.475334 4936 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:12.475507 4936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:12.481748 4936 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:53:12.484910 4970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:12.486515 4936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:12.486801 4936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "f870fa0003ed4c41bbffdd813ed29b13"
format_stamp: "Formatted at 2026-09-28 02:53:09 on dist-test-slave-s1kg"
server_key: "b43805c568de54e353ba0add5d9b9d93"
server_key_iv: "fbf3a1d3d13080d954cb128febd33128"
server_key_version: "encryptionkey@0"
I20260928 02:53:12.487820 4936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:12.510780 4936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:12.514997 4936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:12.515283 4936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:12.516250 4936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:12.517719 4936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:12.517849 4936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:12.517982 4936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:12.518071 4936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:12.552806 4936 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:44193
I20260928 02:53:12.552876 5082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:44193 every 8 connection(s)
I20260928 02:53:12.554335 4936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:12.555747 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 4936
I20260928 02:53:12.556473 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:38083
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:33451
--webserver_port=35795
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:12.573951 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:12.557825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:44807)
0928 02:53:12.558668 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:12.558716 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:12.560680 (+ 1964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:12.562572 (+ 1892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:12.562582 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:12.563242 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:12.563759 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.563776 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.565369 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.565373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.566567 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.566578 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.567679 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.567686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.567955 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.568684 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:12.568713 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:12.572603 (+ 3890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":374,"threads_started":1}
W20260928 02:53:12.573977 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:12.557815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:44807 (local address 127.27.229.254:33451)
0928 02:53:12.561350 (+ 3535us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:12.561361 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:12.561400 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:53:12.561703 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:12.561715 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:12.562053 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:12.562422 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:12.563964 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.565180 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.566719 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.567429 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.567631 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.572972 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:44807: BlockingRecv error: recv got EOF from 127.27.229.193:44807 (error 108)
Metrics: {"server-negotiator.queue_time_us":3334,"thread_start_us":3145,"threads_started":1}
W20260928 02:53:12.574869 5083 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:33451 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.229.254:33451: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:12.583487 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:12.575573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:33629)
0928 02:53:12.575741 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:12.575766 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:12.575897 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:12.576252 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:12.576256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:12.576642 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:12.576774 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.576782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.577780 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.577785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.578670 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.578679 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.579391 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.579396 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.579576 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.580229 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:12.580246 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:12.583130 (+ 2884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:53:12.583590 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:12.575805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:33629 (local address 127.27.229.254:33451)
0928 02:53:12.575998 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:12.576004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:12.576027 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:53:12.576072 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:12.576076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:12.576127 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:12.576226 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:12.576918 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.577642 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.578809 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.579236 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.579381 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.583369 (+ 3988us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:33629: BlockingRecv error: recv got EOF from 127.27.229.193:33629 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:12.592729 5085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:12.584522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:55985)
0928 02:53:12.584702 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:12.584727 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:12.584807 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:12.585129 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:12.585132 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:12.585441 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:12.585581 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.585588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.587029 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.587033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.587925 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:12.587934 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.588840 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.588844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:12.588967 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.589582 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:12.589599 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:12.592455 (+ 2856us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 02:53:12.592959 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:12.584617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:55985 (local address 127.27.229.254:33451)
0928 02:53:12.584803 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:12.584810 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:12.584830 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:53:12.584885 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:12.584890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:12.584971 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:12.585075 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:12.585709 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.586887 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.588047 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:12.588697 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:12.588825 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:12.592700 (+ 3875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:55985: BlockingRecv error: recv got EOF from 127.27.229.193:55985 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 02:53:12.593166 5083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:12.900142 5087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:12.900450 5087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:12.900544 5087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:12.907951 5087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:12.908082 5087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:12.908197 5087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:12.918192 5087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.194.keytab
--principal=oryx/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:38083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=35795
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:12.919790 5087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:12.921156 5087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:12.935039 5095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:12.935077 5092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:12.935110 5093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:12.936208 5087 server_base.cc:1034] running on GCE node
I20260928 02:53:12.936884 5087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:12.937909 5087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:12.939195 5087 hybrid_clock.cc:648] HybridClock initialized: now 1790563992939158 us; error 59 us; skew 500 ppm
I20260928 02:53:12.939548 5087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:12 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563992, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:12.943194 5087 init.cc:377] Logged in from keytab as oryx/127.27.229.194@KRBTEST.COM (short username oryx)
I20260928 02:53:12.944880 5087 webserver.cc:460] Webserver started at http://127.27.229.194:35795/ using document root <none> and password file <none>
I20260928 02:53:12.945598 5087 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:12.945762 5087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:12.951552 5087 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:53:12.959036 5102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:12.960845 5087 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 02:53:12.961050 5087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "5556bf4a99d44a01bf4a6d9a3de6b9fa"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "fcd8f80825b44e9b5c2ec89a5bb52f77"
server_key_iv: "5e7965227c27e42d2d19f509510f9d89"
server_key_version: "encryptionkey@0"
I20260928 02:53:12.962052 5087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:12.984558 5087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:12.989363 5087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:12.989667 5087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:12.990803 5087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:12.992362 5087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:12.992447 5087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:12.992551 5087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:12.992630 5087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:13.028741 5087 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:38083
I20260928 02:53:13.028788 5214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:38083 every 8 connection(s)
I20260928 02:53:13.030114 5087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:13.031927 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5087
I20260928 02:53:13.032652 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:38695
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:33451
--webserver_port=34543
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:53:13.047951 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.032949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:38773 (local address 127.27.229.254:33451)
0928 02:53:13.033129 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.033134 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.034597 (+ 1463us) server_negotiation.cc:407] Connection header received
0928 02:53:13.036725 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.036732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.036836 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.036961 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.038623 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.039482 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.041452 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.041924 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.042075 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.047661 (+ 5586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:38773: BlockingRecv error: recv got EOF from 127.27.229.194:38773 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:53:13.048967 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.033554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:38773)
0928 02:53:13.034401 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.034442 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.036413 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.037218 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.037227 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.037807 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.038437 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.038456 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.039824 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.039830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.041258 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.041273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.042228 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.042239 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.042497 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.043323 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.043354 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.047449 (+ 4095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":353,"threads_started":1}
W20260928 02:53:13.049698 5215 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:33451 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.229.254:33451: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:13.059404 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.050410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:38389)
0928 02:53:13.050721 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.050743 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.050908 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.051285 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.051290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.051671 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.051833 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.051842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.052954 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.052960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.053954 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.053966 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.054806 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.054813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.055040 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.055703 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.055723 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.059087 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179}
W20260928 02:53:13.059620 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.050575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:38389 (local address 127.27.229.254:33451)
0928 02:53:13.050759 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.050768 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.050793 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:53:13.051060 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.051073 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.051137 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.051255 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.051990 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.052790 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.054129 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.054597 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.054768 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.059330 (+ 4562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:38389: BlockingRecv error: recv got EOF from 127.27.229.194:38389 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:13.069339 5217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.060443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:41921)
0928 02:53:13.060700 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.060721 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.060857 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.061205 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.061210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.061550 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.061701 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.061710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.062853 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.062861 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.063907 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.063919 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.064722 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.064729 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.064933 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.065619 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.065687 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.069001 (+ 3314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144}
W20260928 02:53:13.069463 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.060553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:41921 (local address 127.27.229.254:33451)
0928 02:53:13.060725 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.060731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.060780 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:53:13.060991 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.060997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.061070 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.061180 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.061858 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.062672 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.064063 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.064560 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.064718 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.069209 (+ 4491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:41921: BlockingRecv error: recv got EOF from 127.27.229.194:41921 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 02:53:13.069880 5215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:53:13.379136 5218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:13.379496 5218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:13.379671 5218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:13.387529 5218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:13.387694 5218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:13.387897 5218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:13.398411 5218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/krb5kdc/oryx_127.27.229.195.keytab
--principal=oryx/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:38695
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=34543
--tserver_master_addrs=127.27.229.254:33451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:13.399850 5218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:13.401157 5218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:13.414638 5223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:13.414597 5224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:13.414672 5226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:13.415457 5218 server_base.cc:1034] running on GCE node
I20260928 02:53:13.416177 5218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:13.417129 5218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:13.418326 5218 hybrid_clock.cc:648] HybridClock initialized: now 1790563993418285 us; error 43 us; skew 500 ppm
I20260928 02:53:13.418642 5218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790563993, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:13.421914 5218 init.cc:377] Logged in from keytab as oryx/127.27.229.195@KRBTEST.COM (short username oryx)
I20260928 02:53:13.423492 5218 webserver.cc:460] Webserver started at http://127.27.229.195:34543/ using document root <none> and password file <none>
I20260928 02:53:13.424162 5218 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:13.424271 5218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:13.429606 5218 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 02:53:13.432331 5233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:13.433847 5218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:53:13.433988 5218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "13f4b858223e48d7af348c179c059955"
format_stamp: "Formatted at 2026-09-28 02:53:10 on dist-test-slave-s1kg"
server_key: "02b6f986f1b68c3a9809e0c787bff442"
server_key_iv: "3e6101d93e7e5a5e283f9e43eab002c2"
server_key_version: "encryptionkey@0"
I20260928 02:53:13.434983 5218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:13.450774 5218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:13.455881 5218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:13.456125 5218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:13.457269 5218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:13.458565 5218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:13.458650 5218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:13.458724 5218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:13.458775 5218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:13.496582 5218 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:38695
I20260928 02:53:13.496644 5345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:38695 every 8 connection(s)
I20260928 02:53:13.498092 5218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:13.508164 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5218
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:53:13.516446 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.500748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:50965 (local address 127.27.229.254:33451)
0928 02:53:13.500918 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.500923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.502080 (+ 1157us) server_negotiation.cc:407] Connection header received
0928 02:53:13.504109 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.504114 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.504208 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.504315 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.505823 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.506545 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.508631 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.509262 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.509404 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.516098 (+ 6694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:50965: BlockingRecv error: recv got EOF from 127.27.229.195:50965 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:13.516973 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.501223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:50965)
0928 02:53:13.501917 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.501959 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.503834 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.504552 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.504560 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.505084 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.505650 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.505667 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.506713 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.506717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.508465 (+ 1748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.508484 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.509526 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.509536 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.509844 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.510730 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.510762 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.515819 (+ 5057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":278,"threads_started":1}
W20260928 02:53:13.517896 5346 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:33451 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.229.254:33451: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:13.529889 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.518609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:38353)
0928 02:53:13.518947 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.518966 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.519096 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.519457 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.519462 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.519920 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.520061 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.520070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.522108 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.522115 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.523810 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.523829 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.524965 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.524970 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.525111 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.525750 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.525769 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.529526 (+ 3757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":187}
W20260928 02:53:13.530587 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.518724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:38353 (local address 127.27.229.254:33451)
0928 02:53:13.518895 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.518901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.519038 (+ 137us) server_negotiation.cc:407] Connection header received
0928 02:53:13.519233 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.519239 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.519321 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.519432 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.520518 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.521717 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.524011 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.524773 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.524915 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.530208 (+ 5293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:38353: BlockingRecv error: recv got EOF from 127.27.229.195:38353 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260928 02:53:13.533404 5350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.510045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59784 (local address 127.27.229.254:33451)
0928 02:53:13.513188 (+ 3143us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.513195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.513213 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:53:13.513275 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.513280 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.513371 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.513497 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.515644 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.516829 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.518642 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.519361 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.519614 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.523219 (+ 3605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:13.523228 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:13.523247 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:13.523296 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:13.528001 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:13.529034 (+ 1033us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:13.529043 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:13.529052 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:13.529257 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:13.530155 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:13.530162 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:13.530165 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:13.531289 (+ 1124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:13.531921 (+ 632us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:13.532319 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 02:53:13.533033 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":3010,"thread_start_us":2855,"threads_started":1}
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:13.542176 5348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.530768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:40779)
0928 02:53:13.531059 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.531078 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.531196 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.531820 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.531826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.532323 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.532746 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.532768 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.534167 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.534172 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.535346 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.535362 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.536474 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.536481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.537205 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.538203 (+ 998us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.538233 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.541938 (+ 3705us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168}
W20260928 02:53:13.542541 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.530867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:40779 (local address 127.27.229.254:33451)
0928 02:53:13.531249 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.531255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.531278 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:53:13.531336 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.531343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.531439 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.531558 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.532972 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.533993 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.535520 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.536101 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.536275 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.542194 (+ 5919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:40779: BlockingRecv error: recv got EOF from 127.27.229.195:40779 (error 108)
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":519,"threads_started":1}
W20260928 02:53:13.542565 5346 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:13 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:13.603720 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:13.593978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:59057)
0928 02:53:13.594568 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:13.594585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:13.594698 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:13.595110 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:13.595113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:13.595628 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:13.595821 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.595830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.596986 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.596993 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.598206 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:13.598221 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.599016 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.599020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:13.599143 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.599833 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:13.599854 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:13.603491 (+ 3637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":346,"threads_started":1}
W20260928 02:53:13.604454 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:13.594054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:59057 (local address 127.27.229.254:33451)
0928 02:53:13.594253 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:13.594257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:13.594671 (+ 414us) server_negotiation.cc:407] Connection header received
0928 02:53:13.594842 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:13.594847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:13.594958 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:13.595061 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:13.595977 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.596718 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.598382 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:13.598845 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:13.598946 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:13.604288 (+ 5342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:59057: BlockingRecv error: recv got EOF from 127.27.229.193:59057 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:14.079474 5353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:14.070562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:32805)
0928 02:53:14.071158 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:14.071174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:14.071293 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:14.071850 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:14.071854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:14.072380 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:14.072632 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.072649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.073766 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.073772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.074721 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.074733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.075473 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.075479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.075649 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.076254 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:14.076273 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:14.079124 (+ 2851us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":353,"threads_started":1}
W20260928 02:53:14.079596 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:14.070663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:32805 (local address 127.27.229.254:33451)
0928 02:53:14.071037 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:14.071045 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:14.071316 (+ 271us) server_negotiation.cc:407] Connection header received
0928 02:53:14.071399 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:14.071405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:14.071482 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:14.071584 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:14.072791 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.073630 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.074903 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.075293 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.075430 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.079344 (+ 3914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:32805: BlockingRecv error: recv got EOF from 127.27.229.194:32805 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:14.555956 5354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:14.543277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:46053)
0928 02:53:14.543762 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:14.543780 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:14.543898 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:14.544311 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:14.544315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:14.544805 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:14.545030 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.545039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.546091 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.546095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.547345 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.547356 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.548084 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.548090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.548264 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.552406 (+ 4142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:14.552435 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:14.555705 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":282,"threads_started":1}
W20260928 02:53:14.556206 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:14.543450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46053 (local address 127.27.229.254:33451)
0928 02:53:14.543679 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:14.543685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:14.543865 (+ 180us) server_negotiation.cc:407] Connection header received
0928 02:53:14.544061 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:14.544065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:14.544138 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:14.544228 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:14.545205 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.545931 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.547484 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.547899 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.548052 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.555970 (+ 7918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:46053: BlockingRecv error: recv got EOF from 127.27.229.195:46053 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:14 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:14.613508 5355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:14.605012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:54923)
0928 02:53:14.605582 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:14.605598 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:14.605757 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:14.606107 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:14.606111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:14.606537 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:14.606701 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.606710 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.607815 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.607820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.608707 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:14.608717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.609465 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.609471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:14.609635 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.610261 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:14.610279 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:14.613160 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":337,"threads_started":1}
W20260928 02:53:14.613600 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:14.605132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:54923 (local address 127.27.229.254:33451)
0928 02:53:14.605417 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:14.605423 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:14.605684 (+ 261us) server_negotiation.cc:407] Connection header received
0928 02:53:14.605906 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:14.605910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:14.605969 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:14.606075 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:14.606925 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.607668 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.608858 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:14.609311 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:14.609446 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:14.613362 (+ 3916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:54923: BlockingRecv error: recv got EOF from 127.27.229.193:54923 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:15.090498 5356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:15.080766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:58411)
0928 02:53:15.081320 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:15.081336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:15.081459 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:15.081872 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:15.081877 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:15.082408 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:15.082611 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.082622 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.084149 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.084157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.085165 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.085179 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.085981 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.085988 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.086167 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.086854 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:15.086875 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:15.090112 (+ 3237us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":315,"threads_started":1}
W20260928 02:53:15.090569 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:15.080870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:58411 (local address 127.27.229.254:33451)
0928 02:53:15.081068 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:15.081072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:15.081432 (+ 360us) server_negotiation.cc:407] Connection header received
0928 02:53:15.081565 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:15.081569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:15.081672 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:15.081801 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:15.082997 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.083985 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.085341 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.085793 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.085939 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.090349 (+ 4410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:58411: BlockingRecv error: recv got EOF from 127.27.229.194:58411 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:15.566125 5357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:15.556946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:36231)
0928 02:53:15.557566 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:15.557582 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:15.557747 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:15.558107 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:15.558112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:15.558543 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:15.558721 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.558730 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.559914 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.559919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.560852 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.560864 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.561895 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.561899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.562015 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.562724 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:15.562742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:15.565775 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":343,"threads_started":1}
W20260928 02:53:15.566445 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:15.557185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:36231 (local address 127.27.229.254:33451)
0928 02:53:15.557464 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:15.557471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:15.557674 (+ 203us) server_negotiation.cc:407] Connection header received
0928 02:53:15.557907 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:15.557911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:15.557971 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:15.558081 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:15.558919 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.559721 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.560991 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.561715 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.561867 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.566039 (+ 4172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:36231: BlockingRecv error: recv got EOF from 127.27.229.195:36231 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:15 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:15.623670 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:15.614541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:34461)
0928 02:53:15.615073 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:15.615096 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:15.615203 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:15.615636 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:15.615640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:15.615987 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:15.616187 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.616196 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.617274 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.617278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.618210 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:15.618222 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.619120 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.619125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:15.619324 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.619995 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:15.620013 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:15.623389 (+ 3376us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":314,"threads_started":1}
W20260928 02:53:15.624011 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:15.614600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:34461 (local address 127.27.229.254:33451)
0928 02:53:15.614781 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:15.614785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:15.615175 (+ 390us) server_negotiation.cc:407] Connection header received
0928 02:53:15.615383 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:15.615387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:15.615451 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:15.615538 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:15.616380 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.617096 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.618416 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:15.618942 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:15.619039 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:15.623723 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:34461: BlockingRecv error: recv got EOF from 127.27.229.193:34461 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:16.101838 5359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:16.092039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:57219)
0928 02:53:16.092557 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:16.092574 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:16.092808 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:16.093227 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:16.093232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:16.093732 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:16.093954 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.093969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.095160 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.095164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.096088 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.096101 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.097155 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.097160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.097298 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.098335 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:16.098360 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:16.101580 (+ 3220us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":331,"threads_started":1}
W20260928 02:53:16.102103 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:16.092185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:57219 (local address 127.27.229.254:33451)
0928 02:53:16.092371 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:16.092383 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:16.092700 (+ 317us) server_negotiation.cc:407] Connection header received
0928 02:53:16.092973 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:16.092978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:16.093059 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:16.093159 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:16.094091 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.095003 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.096401 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.096934 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.097096 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.101865 (+ 4769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:57219: BlockingRecv error: recv got EOF from 127.27.229.194:57219 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:16.576761 5360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:16.567510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:57587)
0928 02:53:16.568014 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:16.568031 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:16.568197 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:16.568979 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:16.568984 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:16.569484 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:16.569682 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.569690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.570746 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.570751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.571668 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.571679 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.572345 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.572349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.572468 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.573074 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:16.573092 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:16.576498 (+ 3406us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":367,"threads_started":1}
W20260928 02:53:16.577433 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:16.567712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:57587 (local address 127.27.229.254:33451)
0928 02:53:16.567876 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:16.567881 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:16.568187 (+ 306us) server_negotiation.cc:407] Connection header received
0928 02:53:16.568547 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:16.568554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:16.568805 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:16.568905 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:16.569842 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.570579 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.571824 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.572213 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.572320 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.577010 (+ 4690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:57587: BlockingRecv error: recv got EOF from 127.27.229.195:57587 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:16 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:16.634104 5361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:16.624928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:58673)
0928 02:53:16.625430 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:16.625447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:16.625564 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:16.625929 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:16.625932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:16.626296 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:16.626485 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.626493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.627935 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.627939 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.628918 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:16.628929 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.629719 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.629723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:16.629870 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.630589 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:16.630615 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:16.633874 (+ 3259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
W20260928 02:53:16.634379 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:16.624984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:58673 (local address 127.27.229.254:33451)
0928 02:53:16.625222 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:16.625229 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:16.625558 (+ 329us) server_negotiation.cc:407] Connection header received
0928 02:53:16.625718 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:16.625722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:16.625788 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:16.625888 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:16.626747 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.627782 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.629077 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:16.629529 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:16.629631 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:16.634112 (+ 4481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:58673: BlockingRecv error: recv got EOF from 127.27.229.193:58673 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:17.112031 5362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:17.102956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:51895)
0928 02:53:17.103501 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:17.103516 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:17.103626 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:17.103918 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:17.103921 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:17.104274 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:17.104460 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.104468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.105429 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.105433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.106450 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.106462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.107205 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.107210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.107397 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.108163 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:17.108183 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:17.111684 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":298,"threads_started":1}
W20260928 02:53:17.112075 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:17.102999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:51895 (local address 127.27.229.254:33451)
0928 02:53:17.103171 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:17.103175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:17.103590 (+ 415us) server_negotiation.cc:407] Connection header received
0928 02:53:17.103726 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:17.103730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:17.103791 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:17.103875 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:17.104601 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.105298 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.106594 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.107032 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.107127 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.111869 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:51895: BlockingRecv error: recv got EOF from 127.27.229.194:51895 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:17.587052 5363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:17.577816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:35613)
0928 02:53:17.578260 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:17.578276 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:17.578424 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:17.578793 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:17.578797 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:17.579183 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:17.579328 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.579336 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.580454 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.580458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.581984 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.581996 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.582813 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.582817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.582963 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.583665 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:17.583685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:17.586795 (+ 3110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
W20260928 02:53:17.587450 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:17.578003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:35613 (local address 127.27.229.254:33451)
0928 02:53:17.578246 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:17.578255 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:17.578408 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:53:17.578535 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:17.578539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:17.578623 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:17.578719 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:17.579485 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.580275 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.582148 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.582650 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.582753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.587165 (+ 4412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:35613: BlockingRecv error: recv got EOF from 127.27.229.195:35613 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:17 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
W20260928 02:53:17.644433 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:17.635251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:59675 (local address 127.27.229.254:33451)
0928 02:53:17.635495 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:17.635500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:17.635745 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:53:17.635880 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:17.635884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:17.635950 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:17.636044 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:17.636971 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.637784 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.639170 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.639687 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.639791 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.644230 (+ 4439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:59675: BlockingRecv error: recv got EOF from 127.27.229.193:59675 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:53:17.644452 5364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:17.635108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:59675)
0928 02:53:17.635652 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:17.635667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:17.635778 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:17.636125 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:17.636130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:17.636584 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:17.636790 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.636799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.637978 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.637982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.639027 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:17.639047 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:17.639852 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:17.639856 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:17.639973 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:17.640575 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:17.640592 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:17.644026 (+ 3434us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":288,"threads_started":1}
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:18.121701 5365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.112998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:41777)
0928 02:53:18.113533 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:18.113549 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:18.113657 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:18.113947 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:18.113950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:18.114312 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:18.114469 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.114476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.115497 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.115502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.116467 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.116478 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.117343 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.117349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.117527 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.118213 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:18.118238 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:18.121455 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":241,"threads_started":1}
W20260928 02:53:18.121902 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:18.113305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:41777 (local address 127.27.229.254:33451)
0928 02:53:18.113514 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:18.113518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:18.113625 (+ 107us) server_negotiation.cc:407] Connection header received
0928 02:53:18.113761 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:18.113765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:18.113822 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:18.113903 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:18.114584 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.115343 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.116618 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.117173 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.117340 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.121638 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:41777: BlockingRecv error: recv got EOF from 127.27.229.194:41777 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:18.596518 5366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.588019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:47461)
0928 02:53:18.588553 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:18.588569 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:18.588677 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:18.589002 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:18.589005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:18.589399 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:18.589564 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.589573 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.590544 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.590548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.591540 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.591551 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.592399 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.592414 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.592575 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.593266 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:18.593295 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:18.596209 (+ 2914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
W20260928 02:53:18.596725 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:18.588144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:47461 (local address 127.27.229.254:33451)
0928 02:53:18.588357 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:18.588361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:18.588648 (+ 287us) server_negotiation.cc:407] Connection header received
0928 02:53:18.588779 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:18.588783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:18.588854 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:18.588968 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:18.589704 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.590404 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.591657 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.592225 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.592360 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.596467 (+ 4107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:47461: BlockingRecv error: recv got EOF from 127.27.229.195:47461 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:18 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:18.653503 5367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:18.645440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:56635)
0928 02:53:18.645936 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:18.645952 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:18.646064 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:18.646390 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:18.646393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:18.646767 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:18.646958 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.646966 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.648030 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.648033 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.648928 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:18.648938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.649704 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.649708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:18.649833 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.650503 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:18.650520 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:18.653261 (+ 2741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
W20260928 02:53:18.653795 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:18.645530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:56635 (local address 127.27.229.254:33451)
0928 02:53:18.645792 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:18.645799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:18.646032 (+ 233us) server_negotiation.cc:407] Connection header received
0928 02:53:18.646170 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:18.646173 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:18.646233 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:18.646312 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:18.647121 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.647893 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.649068 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:18.649538 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:18.649641 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:18.653527 (+ 3886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:56635: BlockingRecv error: recv got EOF from 127.27.229.193:56635 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:19.131659 5368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.123006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:43421)
0928 02:53:19.123564 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.123581 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.123703 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.124086 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.124090 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.124467 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.124621 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.124630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.125631 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.125636 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.126879 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.126898 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.127904 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.127909 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.128033 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.128658 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.128676 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.131440 (+ 2764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":349,"threads_started":1}
W20260928 02:53:19.131898 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:19.123030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:43421 (local address 127.27.229.254:33451)
0928 02:53:19.123263 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.123272 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.123669 (+ 397us) server_negotiation.cc:407] Connection header received
0928 02:53:19.123847 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.123852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.123914 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.124004 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.124782 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.125478 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.127041 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.127693 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.127814 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.131675 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:43421: BlockingRecv error: recv got EOF from 127.27.229.194:43421 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:19.611270 5369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.597799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:52159)
0928 02:53:19.598676 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.598700 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.598913 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.599330 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.599335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.599787 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.600012 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.600024 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.601454 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.601461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.602747 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.602765 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.603652 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.603657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.603787 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.607550 (+ 3763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.607576 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.611025 (+ 3449us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":433,"threads_started":1}
W20260928 02:53:19.611627 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:19.598089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:52159 (local address 127.27.229.254:33451)
0928 02:53:19.598442 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.598449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.598888 (+ 439us) server_negotiation.cc:407] Connection header received
0928 02:53:19.599090 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.599095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.599181 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.599273 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.600301 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.601293 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.603023 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.603495 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.603598 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.611279 (+ 7681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:52159: BlockingRecv error: recv got EOF from 127.27.229.195:52159 (error 108)
Metrics: {"server-negotiator.queue_time_us":199}
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:19 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:19.663182 5370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:19.654664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:50511)
0928 02:53:19.655306 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:19.655322 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:19.655435 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:19.655729 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:19.655732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:19.656108 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:19.656263 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.656271 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.657275 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.657279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.658188 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:19.658198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.659010 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.659015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:19.659125 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.659737 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:19.659755 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:19.662807 (+ 3052us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":312,"threads_started":1}
W20260928 02:53:19.663314 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:19.654756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:50511 (local address 127.27.229.254:33451)
0928 02:53:19.655045 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:19.655049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:19.655411 (+ 362us) server_negotiation.cc:407] Connection header received
0928 02:53:19.655540 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:19.655544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:19.655603 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:19.655687 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:19.656419 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.657145 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.658326 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:19.658763 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:19.658978 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:19.663087 (+ 4109us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:50511: BlockingRecv error: recv got EOF from 127.27.229.193:50511 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:20.142491 5371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.132789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:36215)
0928 02:53:20.133413 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.133430 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.133544 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.133850 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.133854 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.134241 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:20.134417 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.134426 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.135584 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.135588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.136796 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.136826 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.137718 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.137723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.137859 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.138821 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:20.138872 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:20.142089 (+ 3217us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":301,"threads_started":1}
W20260928 02:53:20.142611 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:20.132864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:36215 (local address 127.27.229.254:33451)
0928 02:53:20.133057 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.133062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.133578 (+ 516us) server_negotiation.cc:407] Connection header received
0928 02:53:20.133633 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.133636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.133710 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.133793 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:20.134585 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.135423 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.137065 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.137483 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.137594 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.142336 (+ 4742us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:36215: BlockingRecv error: recv got EOF from 127.27.229.194:36215 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:20.622028 5372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.612325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:48883)
0928 02:53:20.612910 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.612933 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.613051 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.613620 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.613624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.614174 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:20.614346 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.614355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.615592 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.615596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.616525 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.616537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.617343 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.617348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.617478 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.618148 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:20.618168 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:20.621656 (+ 3488us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":309,"threads_started":1}
W20260928 02:53:20.622165 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:20.612437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:48883 (local address 127.27.229.254:33451)
0928 02:53:20.612670 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.612680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.613052 (+ 372us) server_negotiation.cc:407] Connection header received
0928 02:53:20.613172 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.613176 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.613261 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.613507 (+ 246us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:20.614557 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.615438 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.616678 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.617089 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.617193 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.621945 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:48883: BlockingRecv error: recv got EOF from 127.27.229.195:48883 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:20 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:20.672894 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:20.664207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:33803)
0928 02:53:20.664867 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:20.664888 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:20.665123 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:20.665501 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:20.665505 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:20.665929 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:20.666084 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.666092 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.667162 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.667167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.668068 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:20.668078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.668869 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.668877 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:20.669062 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.669705 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:20.669723 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:20.672667 (+ 2944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":318,"threads_started":1}
W20260928 02:53:20.673292 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:20.664505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:33803 (local address 127.27.229.254:33451)
0928 02:53:20.664742 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:20.664748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:20.665029 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:53:20.665288 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:20.665294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:20.665369 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:20.665473 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:20.666237 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.667022 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.668221 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:20.668680 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:20.668818 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:20.672897 (+ 4079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:33803: BlockingRecv error: recv got EOF from 127.27.229.193:33803 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:21.152305 5374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.143647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:45613)
0928 02:53:21.144182 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:21.144197 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:21.144309 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:21.144657 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:21.144660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:21.145033 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:21.145187 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.145195 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.146431 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.146436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.147414 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.147427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.148194 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.148199 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.148308 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.148958 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:21.148975 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:21.151997 (+ 3022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":288,"threads_started":1}
W20260928 02:53:21.152537 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:21.143687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:45613 (local address 127.27.229.254:33451)
0928 02:53:21.143874 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:21.143878 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:21.144294 (+ 416us) server_negotiation.cc:407] Connection header received
0928 02:53:21.144431 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:21.144435 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:21.144495 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:21.144617 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:21.145304 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.146215 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.147573 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.148013 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.148116 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.152255 (+ 4139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:45613: BlockingRecv error: recv got EOF from 127.27.229.194:45613 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:21.631976 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.623103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:57659)
0928 02:53:21.623709 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:21.623725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:21.623886 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:21.624237 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:21.624242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:21.624649 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:21.624798 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.624806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.625890 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.625895 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.627008 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.627019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.627834 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.627842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.628010 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.628765 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:21.628787 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:21.631729 (+ 2942us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":323,"threads_started":1}
W20260928 02:53:21.632344 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:21.623228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:57659 (local address 127.27.229.254:33451)
0928 02:53:21.623505 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:21.623512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:21.623816 (+ 304us) server_negotiation.cc:407] Connection header received
0928 02:53:21.624035 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:21.624039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:21.624108 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:21.624213 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:21.624939 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.625751 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.627151 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.627593 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.627730 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.631997 (+ 4267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:57659: BlockingRecv error: recv got EOF from 127.27.229.195:57659 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:21 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:21.683548 5376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:21.674010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:57757)
0928 02:53:21.674676 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:21.674693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:21.674807 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:21.675309 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:21.675312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:21.675726 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:21.675888 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.675897 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.677021 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.677029 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.678459 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:21.678473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.679538 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.679543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:21.679688 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.680331 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:21.680390 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:21.683302 (+ 2912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":417,"threads_started":1}
W20260928 02:53:21.683853 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:21.674165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:57757 (local address 127.27.229.254:33451)
0928 02:53:21.674364 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:21.674368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:21.674775 (+ 407us) server_negotiation.cc:407] Connection header received
0928 02:53:21.675089 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:21.675093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:21.675160 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:21.675262 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:21.676038 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.676860 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.678637 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:21.679362 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:21.679516 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:21.683557 (+ 4041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:57757: BlockingRecv error: recv got EOF from 127.27.229.193:57757 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:22.162442 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:22.153595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:33703)
0928 02:53:22.154117 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:22.154133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:22.154291 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:22.154644 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:22.154648 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:22.155112 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:22.155274 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.155282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.156535 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.156541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.157591 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.157602 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.158289 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.158294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.158468 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.159113 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:22.159134 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:22.162194 (+ 3060us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":291,"threads_started":1}
W20260928 02:53:22.162686 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:22.153644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33703 (local address 127.27.229.254:33451)
0928 02:53:22.153881 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:22.153890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:22.154221 (+ 331us) server_negotiation.cc:407] Connection header received
0928 02:53:22.154416 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:22.154429 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:22.154500 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:22.154610 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:22.155415 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.156393 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.157730 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.158131 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.158259 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.162428 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:33703: BlockingRecv error: recv got EOF from 127.27.229.194:33703 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:22.641973 5378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:22.633250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:50685)
0928 02:53:22.633861 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:22.633877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:22.634033 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:22.634387 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:22.634393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:22.634803 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:22.635006 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.635014 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.636040 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.636046 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.636968 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.636978 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.637702 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.637708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.637874 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.638449 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:22.638512 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:22.641627 (+ 3115us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":366,"threads_started":1}
W20260928 02:53:22.642064 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:22.633354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:50685 (local address 127.27.229.254:33451)
0928 02:53:22.633626 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:22.633635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:22.633995 (+ 360us) server_negotiation.cc:407] Connection header received
0928 02:53:22.634165 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:22.634170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:22.634252 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:22.634358 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:22.635150 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.635909 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.637116 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.637547 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.637681 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.641818 (+ 4137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:50685: BlockingRecv error: recv got EOF from 127.27.229.195:50685 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:22 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:22.693231 5379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:22.684549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:59075)
0928 02:53:22.685024 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:22.685040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:22.685149 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:22.685439 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:22.685443 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:22.685805 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:22.685970 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.685979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.687102 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.687106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.688030 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:22.688040 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.688760 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.688764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:22.688873 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.689513 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:22.689533 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:22.692899 (+ 3366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":280,"threads_started":1}
W20260928 02:53:22.693540 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:22.684710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:59075 (local address 127.27.229.254:33451)
0928 02:53:22.684939 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:22.684945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:22.685114 (+ 169us) server_negotiation.cc:407] Connection header received
0928 02:53:22.685244 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:22.685248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:22.685306 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:22.685423 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:22.686109 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.686958 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.688166 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:22.688622 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:22.688716 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:22.693211 (+ 4495us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:59075: BlockingRecv error: recv got EOF from 127.27.229.193:59075 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:23.174544 5380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.163615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:53469)
0928 02:53:23.164117 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.164133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.164246 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.164603 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.164607 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.164982 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.165161 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.165175 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.166404 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.166412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.168185 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.168198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.168992 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.168997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.169158 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.169986 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.170015 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.174142 (+ 4127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":294,"threads_started":1}
W20260928 02:53:23.174652 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:23.163727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:53469 (local address 127.27.229.254:33451)
0928 02:53:23.163975 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.163980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.164236 (+ 256us) server_negotiation.cc:407] Connection header received
0928 02:53:23.164356 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.164360 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.164424 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.164515 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.165337 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.166241 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.168333 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.168830 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.168960 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.174379 (+ 5419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:53469: BlockingRecv error: recv got EOF from 127.27.229.194:53469 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:23.652114 5381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.643249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:51903)
0928 02:53:23.643893 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.643914 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.644053 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.644427 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.644430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.644799 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.644937 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.644945 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.645998 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.646001 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.647107 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.647120 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.647824 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.647828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.647936 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.648676 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.648694 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.651838 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":398,"threads_started":1}
W20260928 02:53:23.652284 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:23.643416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:51903 (local address 127.27.229.254:33451)
0928 02:53:23.643593 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.643602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.644048 (+ 446us) server_negotiation.cc:407] Connection header received
0928 02:53:23.644215 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.644230 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.644297 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.644388 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.645131 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.645858 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.647255 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.647686 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.647787 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.652053 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:51903: BlockingRecv error: recv got EOF from 127.27.229.195:51903 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:23 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:23.702316 5382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:23.694231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:39131)
0928 02:53:23.694793 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:23.694810 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:23.694937 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:23.695234 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:23.695237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:23.695618 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:23.695788 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.695796 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.696775 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.696778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.697675 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:23.697685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.698488 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.698492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:23.698607 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.699248 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:23.699266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:23.702073 (+ 2807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":328,"threads_started":1}
W20260928 02:53:23.702649 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:23.694289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39131 (local address 127.27.229.254:33451)
0928 02:53:23.694513 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:23.694518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:23.694916 (+ 398us) server_negotiation.cc:407] Connection header received
0928 02:53:23.695036 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:23.695039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:23.695114 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:23.695192 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:23.695916 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.696652 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.697835 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:23.698309 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:23.698463 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:23.702282 (+ 3819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:39131: BlockingRecv error: recv got EOF from 127.27.229.193:39131 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:24.184612 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:24.175764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:33975)
0928 02:53:24.176258 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:24.176274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:24.176389 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:24.176745 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:24.176749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:24.177152 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:24.177345 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.177354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.178402 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.178406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.179431 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.179443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.180199 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.180204 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.180339 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.181025 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:24.181044 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:24.184360 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":269,"threads_started":1}
W20260928 02:53:24.184828 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:24.175878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33975 (local address 127.27.229.254:33451)
0928 02:53:24.176137 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:24.176143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:24.176362 (+ 219us) server_negotiation.cc:407] Connection header received
0928 02:53:24.176503 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:24.176507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:24.176574 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:24.176665 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:24.177515 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.178260 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.179601 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.180015 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.180129 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.184547 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:33975: BlockingRecv error: recv got EOF from 127.27.229.194:33975 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:24.664925 5384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:24.653242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:37043)
0928 02:53:24.653800 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:24.653818 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:24.653938 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:24.654253 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:24.654257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:24.654710 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:24.654930 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.654939 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.656015 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.656018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.656971 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.656992 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.657762 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.657767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.657895 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.661550 (+ 3655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:24.661581 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:24.664662 (+ 3081us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":316,"threads_started":1}
W20260928 02:53:24.665148 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:24.653429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:37043 (local address 127.27.229.254:33451)
0928 02:53:24.653605 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:24.653609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:24.653898 (+ 289us) server_negotiation.cc:407] Connection header received
0928 02:53:24.654049 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:24.654052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:24.654113 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:24.654203 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:24.655118 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.655879 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.657137 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.657590 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.657698 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.664878 (+ 7180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:37043: BlockingRecv error: recv got EOF from 127.27.229.195:37043 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:24 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:24.712095 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:24.703427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:51625)
0928 02:53:24.704019 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:24.704035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:24.704145 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:24.704439 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:24.704442 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:24.704808 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:24.704952 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.704960 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.706107 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.706111 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.707132 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:24.707144 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.707981 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.707985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:24.708121 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.708797 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:24.708815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:24.711839 (+ 3024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":324,"threads_started":1}
W20260928 02:53:24.712397 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:24.703481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:51625 (local address 127.27.229.254:33451)
0928 02:53:24.703642 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:24.703646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:24.704111 (+ 465us) server_negotiation.cc:407] Connection header received
0928 02:53:24.704240 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:24.704243 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:24.704303 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:24.704388 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:24.705129 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.705946 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.707279 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:24.707830 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:24.707940 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:24.712100 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:51625: BlockingRecv error: recv got EOF from 127.27.229.193:51625 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:25.194432 5386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:25.185639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:55153)
0928 02:53:25.186172 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:25.186188 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:25.186360 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:25.186688 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:25.186692 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:25.187122 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:25.187266 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.187274 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.188288 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.188296 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.189214 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.189225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.189978 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.189985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.190157 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.190820 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:25.190891 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:25.194098 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":316,"threads_started":1}
W20260928 02:53:25.194578 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:25.185773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:55153 (local address 127.27.229.254:33451)
0928 02:53:25.185991 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:25.185997 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:25.186285 (+ 288us) server_negotiation.cc:407] Connection header received
0928 02:53:25.186488 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:25.186493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:25.186559 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:25.186662 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:25.187415 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.188146 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.189367 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.189806 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.189942 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.194330 (+ 4388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:55153: BlockingRecv error: recv got EOF from 127.27.229.194:55153 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:25.675236 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:25.666041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:48449)
0928 02:53:25.666661 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:25.666676 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:25.666793 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:25.667181 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:25.667185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:25.667608 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:25.667758 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.667766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.668884 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.668888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.669932 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.669946 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.670956 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.670962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.671149 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.671940 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:25.671960 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:25.675000 (+ 3040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":381,"threads_started":1}
W20260928 02:53:25.675455 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:25.666125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:48449 (local address 127.27.229.254:33451)
0928 02:53:25.666394 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:25.666404 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:25.666772 (+ 368us) server_negotiation.cc:407] Connection header received
0928 02:53:25.666922 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:25.666927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:25.666994 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:25.667097 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:25.667915 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.668724 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.670122 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.670735 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.670947 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.675213 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:48449: BlockingRecv error: recv got EOF from 127.27.229.195:48449 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:25 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:25.722150 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:25.713033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:46151)
0928 02:53:25.713604 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:25.713622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:25.713750 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:25.714132 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:25.714136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:25.714527 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:25.714694 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.714702 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.715730 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.715734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.716758 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:25.716770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.717483 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.717489 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:25.717631 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.718265 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:25.718284 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:25.721709 (+ 3425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":307,"threads_started":1}
W20260928 02:53:25.722175 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:25.713118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:46151 (local address 127.27.229.254:33451)
0928 02:53:25.713348 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:25.713358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:25.713712 (+ 354us) server_negotiation.cc:407] Connection header received
0928 02:53:25.713898 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:25.713902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:25.713966 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:25.714068 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:25.714877 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.715597 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.716899 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:25.717304 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:25.717395 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:25.721917 (+ 4522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:46151: BlockingRecv error: recv got EOF from 127.27.229.193:46151 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:26.205892 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:26.196917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:39641)
0928 02:53:26.197545 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:26.197564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:26.197714 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:26.198078 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:26.198083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:26.198607 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:26.198763 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.198771 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.199968 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.199973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.200947 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.200958 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.201779 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.201783 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.201902 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.202531 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:26.202548 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:26.205639 (+ 3091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":310,"threads_started":1}
W20260928 02:53:26.206100 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:26.197131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:39641 (local address 127.27.229.254:33451)
0928 02:53:26.197348 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:26.197354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:26.197677 (+ 323us) server_negotiation.cc:407] Connection header received
0928 02:53:26.197842 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:26.197850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:26.197929 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:26.198015 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:26.198969 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.199820 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.201123 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.201566 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.201712 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.205856 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:39641: BlockingRecv error: recv got EOF from 127.27.229.194:39641 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:26.685211 5390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:26.676219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:41583)
0928 02:53:26.676781 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:26.676799 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:26.676931 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:26.677296 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:26.677300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:26.677749 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:26.677911 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.677920 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.679039 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.679043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.680059 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.680070 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.680923 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.680929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.681077 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.681776 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:26.681796 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:26.684854 (+ 3058us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":271,"threads_started":1}
W20260928 02:53:26.685315 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:26.676432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:41583 (local address 127.27.229.254:33451)
0928 02:53:26.676654 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:26.676662 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:26.676913 (+ 251us) server_negotiation.cc:407] Connection header received
0928 02:53:26.677084 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:26.677088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:26.677151 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:26.677241 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:26.678085 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.678899 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.680229 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.680714 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.680816 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.685080 (+ 4264us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:41583: BlockingRecv error: recv got EOF from 127.27.229.195:41583 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:26 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:26.731256 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:26.723044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:36243)
0928 02:53:26.723644 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:26.723661 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:26.723849 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:26.724288 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:26.724291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:26.724652 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:26.724828 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.724838 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.725842 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.725845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.726740 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:26.726750 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.727483 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.727487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:26.727611 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.728260 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:26.728280 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:26.731002 (+ 2722us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":329,"threads_started":1}
W20260928 02:53:26.731726 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:26.723127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:36243 (local address 127.27.229.254:33451)
0928 02:53:26.723500 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:26.723508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:26.723752 (+ 244us) server_negotiation.cc:407] Connection header received
0928 02:53:26.724010 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:26.724015 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:26.724117 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:26.724232 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:26.724976 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.725722 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.726921 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:26.727355 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:26.727458 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:26.731236 (+ 3778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:36243: BlockingRecv error: recv got EOF from 127.27.229.193:36243 (error 108)
Metrics: {"server-negotiator.queue_time_us":216}
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:27.215863 5392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.206954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:39055)
0928 02:53:27.207484 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:27.207503 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:27.207668 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:27.208024 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:27.208027 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:27.208395 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:27.208618 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.208628 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.209705 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.209709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.210695 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.210705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.211494 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.211499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.211647 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.212276 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:27.212294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:27.215540 (+ 3246us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":298,"threads_started":1}
W20260928 02:53:27.216066 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:27.207059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:39055 (local address 127.27.229.254:33451)
0928 02:53:27.207227 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:27.207231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:27.207618 (+ 387us) server_negotiation.cc:407] Connection header received
0928 02:53:27.207814 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:27.207818 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:27.207892 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:27.207981 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:27.208839 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.209573 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.210926 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.211324 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.211465 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.215839 (+ 4374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:39055: BlockingRecv error: recv got EOF from 127.27.229.194:39055 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.195@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:27.695094 5393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.686233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.195:56683)
0928 02:53:27.686823 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:27.686864 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:27.686985 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:27.687346 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:27.687350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:27.687782 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:27.687976 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.687986 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.689053 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.689057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.690095 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.690108 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.690895 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.690904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.691109 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.691839 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:27.691856 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:27.694789 (+ 2933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":340,"threads_started":1}
W20260928 02:53:27.695329 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:27.686391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:56683 (local address 127.27.229.254:33451)
0928 02:53:27.686594 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:27.686598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:27.686950 (+ 352us) server_negotiation.cc:407] Connection header received
0928 02:53:27.687093 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:27.687097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:27.687171 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:27.687263 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:27.688128 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.688904 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.690239 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.690640 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.690741 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.695096 (+ 4355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.195:56683: BlockingRecv error: recv got EOF from 127.27.229.195:56683 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:27 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.193@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:27.741590 5394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:27.732360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.193:38285)
0928 02:53:27.732903 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:27.732921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:27.733043 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:27.733359 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:27.733363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:27.733809 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:27.733986 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.733995 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.735007 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.735011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.736066 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:27.736079 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.736795 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.736799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:27.736919 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.737736 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:27.737766 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:27.741202 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":304,"threads_started":1}
W20260928 02:53:27.741686 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:27.732422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:38285 (local address 127.27.229.254:33451)
0928 02:53:27.732670 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:27.732681 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:27.733004 (+ 323us) server_negotiation.cc:407] Connection header received
0928 02:53:27.733168 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:27.733172 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:27.733230 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:27.733312 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:27.734119 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.734870 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.736217 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:27.736650 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:27.736749 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:27.741433 (+ 4684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.193:38285: BlockingRecv error: recv got EOF from 127.27.229.193:38285 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 28 02:53:28 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
Sep 28 02:53:28 dist-test-slave-s1kg krb5kdc[4348](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.229.194@KRBTEST.COM for oryx/127.27.229.254@KRBTEST.COM, Server not found in Kerberos database
I20260928 02:53:28.226183 5395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.217214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33451 (local address 127.27.229.194:44287)
0928 02:53:28.217779 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:28.217795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:28.217908 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:28.218310 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:28.218315 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:28.218929 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:28.219177 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.219188 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.220300 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.220304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.221206 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:28.221223 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.222011 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.222015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:28.222140 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.222875 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:28.222928 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:28.225918 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.229.254:33451: Server oryx/127.27.229.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":311,"threads_started":1}
W20260928 02:53:28.226387 5086 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:28.217303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:44287 (local address 127.27.229.254:33451)
0928 02:53:28.217487 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.217500 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.217887 (+ 387us) server_negotiation.cc:407] Connection header received
0928 02:53:28.218041 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.218044 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.218138 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.218241 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.219304 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.220112 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.221343 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.221872 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.221972 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.226155 (+ 4183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.229.194:44287: BlockingRecv error: recv got EOF from 127.27.229.194:44287 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:53:28.517385 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4936
I20260928 02:53:28.530628 5397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.518633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56370 (local address 127.27.229.193:44193)
0928 02:53:28.519270 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.519278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.519305 (+ 27us) server_negotiation.cc:407] Connection header received
0928 02:53:28.519364 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.519370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.519651 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.519815 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.521275 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.522444 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.523834 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.524387 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.524507 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.527202 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.527212 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.527226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.527261 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.530071 (+ 2810us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:53:28.530357 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56370: Request ticket server kudu/127.27.229.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":367,"threads_started":1}
W20260928 02:53:28.530864 5397 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56370: Request ticket server kudu/127.27.229.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:53:28.530869 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4936: Not authorized: Client connection negotiation failed: client connection to 127.27.229.193:44193: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:53:28.531122 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4936
I20260928 02:53:28.548425 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5087
I20260928 02:53:28.561367 5398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.549773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36688 (local address 127.27.229.194:38083)
0928 02:53:28.550445 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.550452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.550484 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:53:28.550547 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.550554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.550784 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.551039 (+ 255us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.552594 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.553661 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555027 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.555645 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.555761 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.558249 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.558257 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.558270 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.558305 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.560860 (+ 2555us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:53:28.561138 (+ 278us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36688: Request ticket server kudu/127.27.229.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":326,"threads_started":1}
W20260928 02:53:28.561484 5398 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36688: Request ticket server kudu/127.27.229.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:53:28.561637 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5087: Not authorized: Client connection negotiation failed: client connection to 127.27.229.194:38083: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:53:28.561797 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5087
I20260928 02:53:28.579721 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5218
I20260928 02:53:28.592413 5399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:28.580812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.27.229.195:38695)
0928 02:53:28.581507 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:28.581514 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:28.581547 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:53:28.581607 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:28.581613 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:28.581826 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:28.581977 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:28.583491 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.584569 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.585839 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:28.586403 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:28.586509 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:28.589258 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:28.589268 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:28.589283 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:28.589318 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:28.591905 (+ 2587us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0928 02:53:28.592178 (+ 273us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56170: Request ticket server kudu/127.27.229.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":391,"threads_started":1}
W20260928 02:53:28.592538 5399 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56170: Request ticket server kudu/127.27.229.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260928 02:53:28.592584 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5218: Not authorized: Client connection negotiation failed: client connection to 127.27.229.195:38695: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.27.229.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260928 02:53:28.592748 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5218
I20260928 02:53:28.609104 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 4867
I20260928 02:53:28.610428 4929 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:28.727782 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 4867
2026-09-28T02:53:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21366 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T02:53:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:28Z Disabled control of system clock
I20260928 02:53:28.807850 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46157
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 02:53:29.139091 5404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:29.139381 5404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:29.139461 5404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:29.146620 5404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:29.146732 5404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:29.146770 5404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:29.146818 5404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:29.146938 5404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:29.156095 5404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46157
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:29.157351 5404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:29.159164 5404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:29.171293 5409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.171257 5410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.171322 5412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.171948 5404 server_base.cc:1034] running on GCE node
I20260928 02:53:29.172725 5404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:29.174089 5404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:29.175267 5404 hybrid_clock.cc:648] HybridClock initialized: now 1790564009175216 us; error 50 us; skew 500 ppm
I20260928 02:53:29.175776 5404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:29.178133 5404 webserver.cc:460] Webserver started at http://127.27.229.254:34271/ using document root <none> and password file <none>
I20260928 02:53:29.178947 5404 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:29.179056 5404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:29.179388 5404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:29.182273 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "49503a5a7f8544e995a29083147784f3"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "949535e36281ac55b82cf08c3e9c5373"
server_key_iv: "46caae4b0f44be4ed447eaee80cb7e16"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.183164 5404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "49503a5a7f8544e995a29083147784f3"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "949535e36281ac55b82cf08c3e9c5373"
server_key_iv: "46caae4b0f44be4ed447eaee80cb7e16"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.188486 5404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:53:29.192159 5418 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.193678 5404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:29.193871 5404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "49503a5a7f8544e995a29083147784f3"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "949535e36281ac55b82cf08c3e9c5373"
server_key_iv: "46caae4b0f44be4ed447eaee80cb7e16"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.194121 5404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:29.209592 5404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:29.210511 5404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:29.210937 5404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:29.230181 5404 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:46157
I20260928 02:53:29.230201 5469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:46157 every 8 connection(s)
I20260928 02:53:29.231705 5404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:29.235600 5470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:29.240340 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5404
I20260928 02:53:29.240548 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:53:29.241367 28567 external_mini_cluster.cc:1428] Setting key bebf1fc948ab867f9206daa614b67959
I20260928 02:53:29.247526 5470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3: Bootstrap starting.
I20260928 02:53:29.251029 5470 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:29.252118 5470 log.cc:824] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:29.255731 5470 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3: No bootstrap required, opened a new log
I20260928 02:53:29.256054 5473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.243544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38218 (local address 127.27.229.254:46157)
0928 02:53:29.244815 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.244830 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.244884 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:53:29.246546 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.246569 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.247130 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.247530 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.248178 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.249555 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.250936 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.251721 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.251931 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.253263 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.253271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.253286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:29.253942 (+ 656us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.254054 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:29.254081 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.254208 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.254445 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.254771 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":515,"threads_started":1}
I20260928 02:53:29.263926 5470 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } }
I20260928 02:53:29.264281 5470 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:29.264353 5470 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49503a5a7f8544e995a29083147784f3, State: Initialized, Role: FOLLOWER
I20260928 02:53:29.264850 5470 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } }
I20260928 02:53:29.265067 5470 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:29.265244 5470 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:29.265437 5470 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:29.267745 5470 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } }
I20260928 02:53:29.268271 5470 leader_election.cc:302] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49503a5a7f8544e995a29083147784f3; no voters:
I20260928 02:53:29.268860 5470 leader_election.cc:288] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:29.269039 5475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:29.269743 5475 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [term 1 LEADER]: Becoming Leader. State: Replica: 49503a5a7f8544e995a29083147784f3, State: Running, Role: LEADER
I20260928 02:53:29.270305 5475 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } }
I20260928 02:53:29.271056 5470 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:29.273232 5477 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49503a5a7f8544e995a29083147784f3. Latest consensus state: current_term: 1 leader_uuid: "49503a5a7f8544e995a29083147784f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } } }
I20260928 02:53:29.273604 5477 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:29.274606 5476 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49503a5a7f8544e995a29083147784f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49503a5a7f8544e995a29083147784f3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46157 } } }
I20260928 02:53:29.275098 5476 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:29.277213 5484 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:29.282534 5484 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:29.291635 5484 catalog_manager.cc:1347] Generated new cluster ID: 89cf282c956c40b49f26cc647a88ba68
I20260928 02:53:29.291755 5484 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:29.303092 5484 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:29.304626 5484 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:29.312610 5484 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49503a5a7f8544e995a29083147784f3: Generated new TSK 0
I20260928 02:53:29.313462 5484 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:53:29.318358 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--builtin_ntp_servers=127.27.229.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 02:53:29.653405 5494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:29.653712 5494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:29.653851 5494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:29.661751 5494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:29.661880 5494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:29.662052 5494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:29.672271 5494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:29.673599 5494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:29.675469 5494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:29.689348 5499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.689365 5502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:29.689577 5500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:29.690213 5494 server_base.cc:1034] running on GCE node
I20260928 02:53:29.690974 5494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:29.692037 5494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:29.693290 5494 hybrid_clock.cc:648] HybridClock initialized: now 1790564009693220 us; error 68 us; skew 500 ppm
I20260928 02:53:29.693645 5494 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:29.700723 5494 webserver.cc:460] Webserver started at http://127.27.229.193:45927/ using document root <none> and password file <none>
I20260928 02:53:29.701563 5494 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:29.701702 5494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:29.702052 5494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:29.705044 5494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "f9bce3c0f6ac4642b6d7d17bab01dc5a"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "f067515abfd5ff95738a64df6fe4b03e"
server_key_iv: "57ede4705489c123343c0639a67da11b"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.705884 5494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "f9bce3c0f6ac4642b6d7d17bab01dc5a"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "f067515abfd5ff95738a64df6fe4b03e"
server_key_iv: "57ede4705489c123343c0639a67da11b"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.711364 5494 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:53:29.715086 5508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.716675 5494 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260928 02:53:29.716902 5494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "f9bce3c0f6ac4642b6d7d17bab01dc5a"
format_stamp: "Formatted at 2026-09-28 02:53:29 on dist-test-slave-s1kg"
server_key: "f067515abfd5ff95738a64df6fe4b03e"
server_key_iv: "57ede4705489c123343c0639a67da11b"
server_key_version: "encryptionkey@0"
I20260928 02:53:29.717161 5494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:29.754034 5494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:29.755100 5494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:29.755431 5494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:29.756491 5494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:29.758163 5494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:29.758302 5494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.758446 5494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:29.758553 5494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:29.795674 5494 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39569
I20260928 02:53:29.795770 5620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39569 every 8 connection(s)
I20260928 02:53:29.797118 5494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:29.801859 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5494
I20260928 02:53:29.802107 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:29.802780 28567 external_mini_cluster.cc:1428] Setting key da4d7b7095ffd5bf59a04ef545ce9a14
I20260928 02:53:29.805730 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--builtin_ntp_servers=127.27.229.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:53:29.811414 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.800075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:54185 (local address 127.27.229.254:46157)
0928 02:53:29.800535 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:29.800540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:29.801685 (+ 1145us) server_negotiation.cc:407] Connection header received
0928 02:53:29.803664 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:29.803670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:29.803765 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:29.803886 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:29.805122 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.806260 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.808056 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.808765 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.808903 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.810388 (+ 1485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:29.810397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:29.810413 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:29.810438 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:29.810490 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:29.810507 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:29.810599 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:29.810856 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 02:53:29.810960 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":251,"threads_started":1}
I20260928 02:53:29.812539 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:29.800583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46157 (local address 127.27.229.193:54185)
0928 02:53:29.801510 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:29.801552 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:29.803395 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:29.804101 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:29.804110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:29.804211 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:29.804943 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.804962 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.806422 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.806427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.807899 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:29.807912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:29.809055 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:29.809062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:29.809285 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:29.810056 (+ 771us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:29.810059 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:29.810102 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:29.810185 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:29.810613 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:29.810621 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:29.810628 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:53:29.810790 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:53:29.811189 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":323,"threads_started":1}
I20260928 02:53:29.814248 5621 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46157
I20260928 02:53:29.814566 5621 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:29.815411 5621 heartbeater.cc:511] Master 127.27.229.254:46157 requested a full tablet report, sending...
I20260928 02:53:29.817368 5435 ts_manager.cc:194] Registered new tserver with Master: f9bce3c0f6ac4642b6d7d17bab01dc5a (127.27.229.193:39569)
I20260928 02:53:29.818759 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.193:54185
W20260928 02:53:30.132910 5625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.133289 5625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.133450 5625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.140756 5625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.140926 5625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.141119 5625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:30.150310 5625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:30.151672 5625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.153399 5625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.166355 5633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.166420 5631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.166404 5630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.167150 5625 server_base.cc:1034] running on GCE node
I20260928 02:53:30.167724 5625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.168658 5625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.169904 5625 hybrid_clock.cc:648] HybridClock initialized: now 1790564010169841 us; error 62 us; skew 500 ppm
I20260928 02:53:30.170241 5625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:30.172771 5625 webserver.cc:460] Webserver started at http://127.27.229.194:38283/ using document root <none> and password file <none>
I20260928 02:53:30.173465 5625 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.173594 5625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.173913 5625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:30.177047 5625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "7013d526424542eaabe1d359eff142d9"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "ee5f8a6677394b95df6e5e9a58305cdb"
server_key_iv: "208200cbd14589cc6fae2bd07c30abc7"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.177834 5625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "7013d526424542eaabe1d359eff142d9"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "ee5f8a6677394b95df6e5e9a58305cdb"
server_key_iv: "208200cbd14589cc6fae2bd07c30abc7"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.183069 5625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:53:30.186564 5639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.188055 5625 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:30.188264 5625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "7013d526424542eaabe1d359eff142d9"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "ee5f8a6677394b95df6e5e9a58305cdb"
server_key_iv: "208200cbd14589cc6fae2bd07c30abc7"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.188500 5625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.214192 5625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.215068 5625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.215380 5625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.216428 5625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:30.217876 5625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:30.217999 5625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.218138 5625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:30.218233 5625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.253209 5625 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:46095
I20260928 02:53:30.253299 5751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:46095 every 8 connection(s)
I20260928 02:53:30.254492 5625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:30.261096 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5625
I20260928 02:53:30.261270 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:30.261802 28567 external_mini_cluster.cc:1428] Setting key c475a04c5d1361bff54474b0721a76f1
I20260928 02:53:30.265254 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--builtin_ntp_servers=127.27.229.212:41581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 02:53:30.269404 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.256996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:43299 (local address 127.27.229.254:46157)
0928 02:53:30.257173 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.257177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.258483 (+ 1306us) server_negotiation.cc:407] Connection header received
0928 02:53:30.260322 (+ 1839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.260326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.260399 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.260517 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.262077 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.262997 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.265582 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.266329 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.266456 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.268603 (+ 2147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:30.268608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:30.268611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:30.268631 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:30.268663 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:30.268675 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:30.268757 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:30.269071 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:53:30.269166 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:53:30.271059 5754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.257451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46157 (local address 127.27.229.194:43299)
0928 02:53:30.258315 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.258350 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:30.260025 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:30.260868 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:30.260883 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:30.261015 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:30.261867 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:30.261896 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.263165 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.263171 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:30.265373 (+ 2202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:30.265393 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.266885 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.266895 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:30.267255 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.268185 (+ 930us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:30.268191 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:30.268253 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:30.268380 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:30.268786 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:30.268799 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:30.268809 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:53:30.269011 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 02:53:30.269487 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":262,"threads_started":1}
I20260928 02:53:30.272822 5752 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46157
I20260928 02:53:30.273200 5752 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:30.274044 5752 heartbeater.cc:511] Master 127.27.229.254:46157 requested a full tablet report, sending...
I20260928 02:53:30.275739 5435 ts_manager.cc:194] Registered new tserver with Master: 7013d526424542eaabe1d359eff142d9 (127.27.229.194:46095)
I20260928 02:53:30.277038 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.194:43299
W20260928 02:53:30.604929 5755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:30.605217 5755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:30.605297 5755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:30.612661 5755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:30.612771 5755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:30.612874 5755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:30.622334 5755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:41581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:30.623632 5755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:30.625355 5755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:30.638818 5760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.638931 5763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:30.638906 5761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:30.639701 5755 server_base.cc:1034] running on GCE node
I20260928 02:53:30.640304 5755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:30.641254 5755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:30.642450 5755 hybrid_clock.cc:648] HybridClock initialized: now 1790564010642406 us; error 46 us; skew 500 ppm
I20260928 02:53:30.642814 5755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:30.645263 5755 webserver.cc:460] Webserver started at http://127.27.229.195:39889/ using document root <none> and password file <none>
I20260928 02:53:30.646028 5755 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:30.646118 5755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:30.646546 5755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:30.649569 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "eef5bfb03cef4bfea2e1bb0b5b993b60"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "af34bc4a012fcfc190995c5c3f8eb551"
server_key_iv: "a7362a1493c169557ad51f8d857c25b9"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.650391 5755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "eef5bfb03cef4bfea2e1bb0b5b993b60"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "af34bc4a012fcfc190995c5c3f8eb551"
server_key_iv: "a7362a1493c169557ad51f8d857c25b9"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.656081 5755 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:53:30.659900 5769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.661470 5755 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:53:30.661667 5755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "eef5bfb03cef4bfea2e1bb0b5b993b60"
format_stamp: "Formatted at 2026-09-28 02:53:30 on dist-test-slave-s1kg"
server_key: "af34bc4a012fcfc190995c5c3f8eb551"
server_key_iv: "a7362a1493c169557ad51f8d857c25b9"
server_key_version: "encryptionkey@0"
I20260928 02:53:30.661913 5755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:30.688627 5755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:30.689497 5755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:30.689814 5755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:30.690805 5755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:30.692430 5755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:30.692520 5755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.692677 5755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:30.692749 5755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:30.729012 5755 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45227
I20260928 02:53:30.729120 5881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45227 every 8 connection(s)
I20260928 02:53:30.730440 5755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:30.732414 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5755
I20260928 02:53:30.732589 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:30.733075 28567 external_mini_cluster.cc:1428] Setting key 851e96602b05e5ebbab3767615a49f7b
I20260928 02:53:30.743963 5623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.733303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:55619 (local address 127.27.229.254:46157)
0928 02:53:30.733538 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.733545 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.735413 (+ 1868us) server_negotiation.cc:407] Connection header received
0928 02:53:30.737285 (+ 1872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.737289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.737355 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.737439 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.738560 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.739582 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.741260 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.741650 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.741742 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.743255 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:30.743260 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:30.743264 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:30.743284 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:30.743313 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:30.743325 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:30.743409 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:30.743630 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 02:53:30.743713 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:53:30.745038 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.733950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46157 (local address 127.27.229.195:55619)
0928 02:53:30.735240 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:30.735295 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:30.737015 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:30.737674 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:30.737682 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:30.737782 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:30.738386 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:30.738403 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.739738 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.739743 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:30.741129 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:30.741141 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.741856 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.741861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:30.742083 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.742804 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:30.742807 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:30.742932 (+ 125us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:30.743025 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:30.743430 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:30.743438 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:30.743443 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 02:53:30.743604 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 02:53:30.743932 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":435,"threads_started":1}
I20260928 02:53:30.746789 5882 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46157
I20260928 02:53:30.747269 5882 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:30.748255 5882 heartbeater.cc:511] Master 127.27.229.254:46157 requested a full tablet report, sending...
I20260928 02:53:30.749769 5435 ts_manager.cc:194] Registered new tserver with Master: eef5bfb03cef4bfea2e1bb0b5b993b60 (127.27.229.195:45227)
I20260928 02:53:30.750593 5435 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.195:55619
I20260928 02:53:30.761576 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:53:30.768339 5623 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:30.766733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.27.229.254:46157)
0928 02:53:30.766958 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.766962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.767453 (+ 491us) server_negotiation.cc:407] Connection header received
0928 02:53:30.767528 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.767532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.767600 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.767694 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.768041 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38234: BlockingRecv error: recv got EOF from 127.0.0.1:38234 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:53:30.769104 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5494
I20260928 02:53:30.777743 5891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.770041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48540 (local address 127.27.229.193:39569)
0928 02:53:30.770568 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.770575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.770604 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:53:30.770661 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.770667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.770947 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.771145 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.771584 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.772732 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.774101 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.774662 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.774769 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.776160 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:30.776170 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:30.776183 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:30.776801 (+ 618us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:30.776860 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:30.776880 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:30.776983 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:30.777300 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:53:30.777424 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":270,"threads_started":1}
I20260928 02:53:30.779136 5616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:30.860136 5621 heartbeater.cc:503] Master 127.27.229.254:46157 was elected leader, sending a full tablet report...
I20260928 02:53:30.861289 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5494
W20260928 02:53:30.879820 5420 connection.cc:535] server connection from 127.27.229.193:54185 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:53:30.880151 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5625
I20260928 02:53:30.888216 5893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.881134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57822 (local address 127.27.229.194:46095)
0928 02:53:30.881608 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.881623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.881651 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:53:30.881711 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.881717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.881930 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.882066 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.882455 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.883575 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.884841 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.885447 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.885557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.886622 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:30.886629 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:30.886643 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:30.887341 (+ 698us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:30.887387 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:30.887407 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:30.887502 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:30.887677 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:53:30.887827 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":232,"threads_started":1}
I20260928 02:53:30.889428 5747 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:30.971769 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5625
I20260928 02:53:30.988554 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5755
I20260928 02:53:30.996927 5895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:30.989634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42956 (local address 127.27.229.195:45227)
0928 02:53:30.990226 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:30.990233 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:30.990263 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:53:30.990322 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:30.990328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:30.990559 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:30.990733 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:30.991220 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.992242 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.993469 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:30.994031 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:30.994176 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:30.995388 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:30.995397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:30.995411 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:30.996020 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:30.996065 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:30.996085 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:30.996231 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:30.996412 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 02:53:30.996542 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":298,"threads_started":1}
I20260928 02:53:30.998386 5877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:31.081349 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5755
I20260928 02:53:31.099282 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5404
I20260928 02:53:31.100616 5465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:31.202562 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5404
2026-09-28T02:53:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2467 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T02:53:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:31Z Disabled control of system clock
I20260928 02:53:31.275179 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37853
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:37853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:53:31.607659 5901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:31.607954 5901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:31.608042 5901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:31.615326 5901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:31.615445 5901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:31.615484 5901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:31.615540 5901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:31.615585 5901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:31.624783 5901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:37853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:31.626045 5901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:31.627833 5901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:31.639685 5909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:31.639725 5907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:31.639783 5906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:31.640630 5901 server_base.cc:1034] running on GCE node
I20260928 02:53:31.641425 5901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:31.642911 5901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:31.644155 5901 hybrid_clock.cc:648] HybridClock initialized: now 1790564011644100 us; error 46 us; skew 500 ppm
I20260928 02:53:31.644524 5901 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:31.647262 5901 webserver.cc:460] Webserver started at http://127.27.229.254:33719/ using document root <none> and password file <none>
I20260928 02:53:31.647986 5901 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:31.648113 5901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:31.648448 5901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:31.651405 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "66ee84a920a346e1a400d1deb0330586"
format_stamp: "Formatted at 2026-09-28 02:53:31 on dist-test-slave-s1kg"
server_key: "77a77af2b0593d979bcf0abf90ed96ab"
server_key_iv: "53a49af89fec36c4b7ec6465daf0f801"
server_key_version: "encryptionkey@0"
I20260928 02:53:31.652222 5901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "66ee84a920a346e1a400d1deb0330586"
format_stamp: "Formatted at 2026-09-28 02:53:31 on dist-test-slave-s1kg"
server_key: "77a77af2b0593d979bcf0abf90ed96ab"
server_key_iv: "53a49af89fec36c4b7ec6465daf0f801"
server_key_version: "encryptionkey@0"
I20260928 02:53:31.657496 5901 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:53:31.661000 5915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:31.662308 5901 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:31.662494 5901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "66ee84a920a346e1a400d1deb0330586"
format_stamp: "Formatted at 2026-09-28 02:53:31 on dist-test-slave-s1kg"
server_key: "77a77af2b0593d979bcf0abf90ed96ab"
server_key_iv: "53a49af89fec36c4b7ec6465daf0f801"
server_key_version: "encryptionkey@0"
I20260928 02:53:31.662715 5901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:31.668434 5901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:31.669221 5901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:31.669518 5901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:31.688076 5901 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:37853
I20260928 02:53:31.688163 5966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:37853 every 8 connection(s)
I20260928 02:53:31.689424 5901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:31.693135 5967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:31.698235 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5901
I20260928 02:53:31.698455 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:53:31.699237 28567 external_mini_cluster.cc:1428] Setting key 5d8d50d89a7317bdb1e52095bac7bc81
I20260928 02:53:31.705142 5967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586: Bootstrap starting.
I20260928 02:53:31.709164 5967 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:31.709604 5970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:31.701810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49344 (local address 127.27.229.254:37853)
0928 02:53:31.703141 (+ 1331us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:31.703156 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:31.703210 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:53:31.705599 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:31.705626 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:31.706146 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:31.706577 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:31.707668 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:31.707678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:31.707696 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:31.707712 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:31.707786 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:31.707805 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:31.708019 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:31.708137 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:53:31.708392 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":424,"threads_started":1}
I20260928 02:53:31.710409 5967 log.cc:824] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:31.713660 5967 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586: No bootstrap required, opened a new log
I20260928 02:53:31.720750 5967 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } }
I20260928 02:53:31.721076 5967 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:31.721201 5967 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66ee84a920a346e1a400d1deb0330586, State: Initialized, Role: FOLLOWER
I20260928 02:53:31.721812 5967 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } }
I20260928 02:53:31.722069 5967 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:31.722234 5967 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:31.722424 5967 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:31.724615 5967 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } }
I20260928 02:53:31.725111 5967 leader_election.cc:302] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66ee84a920a346e1a400d1deb0330586; no voters:
I20260928 02:53:31.725687 5967 leader_election.cc:288] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:31.725842 5972 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:31.726948 5972 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [term 1 LEADER]: Becoming Leader. State: Replica: 66ee84a920a346e1a400d1deb0330586, State: Running, Role: LEADER
I20260928 02:53:31.727587 5972 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } }
I20260928 02:53:31.728228 5967 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:31.730593 5973 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66ee84a920a346e1a400d1deb0330586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } } }
I20260928 02:53:31.730569 5974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66ee84a920a346e1a400d1deb0330586. Latest consensus state: current_term: 1 leader_uuid: "66ee84a920a346e1a400d1deb0330586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66ee84a920a346e1a400d1deb0330586" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37853 } } }
I20260928 02:53:31.731033 5973 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:31.731106 5974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586 [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:31.732492 5980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:31.738250 5980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:31.747223 5980 catalog_manager.cc:1347] Generated new cluster ID: e9fd9149d25c452c9673fbf3f4a952a5
I20260928 02:53:31.747349 5980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:31.763562 5980 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:31.764025 5980 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:31.771293 5980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66ee84a920a346e1a400d1deb0330586: Generated new TSK 0
I20260928 02:53:31.772390 5980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:53:31.778342 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--builtin_ntp_servers=127.27.229.212:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 02:53:32.118752 5991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:32.119123 5991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:32.119215 5991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:32.126610 5991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:32.126727 5991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:32.126861 5991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:32.136478 5991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:32.137884 5991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:32.139736 5991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:32.153244 5996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:32.153211 5997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:32.153291 5999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:32.154647 5991 server_base.cc:1034] running on GCE node
I20260928 02:53:32.155325 5991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:32.156354 5991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:32.157632 5991 hybrid_clock.cc:648] HybridClock initialized: now 1790564012157575 us; error 51 us; skew 500 ppm
I20260928 02:53:32.157982 5991 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:32.160534 5991 webserver.cc:460] Webserver started at http://127.27.229.193:42749/ using document root <none> and password file <none>
I20260928 02:53:32.161252 5991 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:32.161379 5991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:32.161717 5991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:32.166023 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "8bdf140f045e4c65ab2652bd8bc75174"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "056b103f9fd55bef53aad42c3919ce88"
server_key_iv: "149a9f1d3e0f44d2475c52b15b125741"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.166997 5991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "8bdf140f045e4c65ab2652bd8bc75174"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "056b103f9fd55bef53aad42c3919ce88"
server_key_iv: "149a9f1d3e0f44d2475c52b15b125741"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.172287 5991 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:53:32.175935 6005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.177389 5991 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:53:32.177606 5991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "8bdf140f045e4c65ab2652bd8bc75174"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "056b103f9fd55bef53aad42c3919ce88"
server_key_iv: "149a9f1d3e0f44d2475c52b15b125741"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.177866 5991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:32.184312 5991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:32.185207 5991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:32.185518 5991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:32.186530 5991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:32.188081 5991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:32.188215 5991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.188359 5991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:32.188462 5991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.223556 5991 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37083
I20260928 02:53:32.223582 6117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37083 every 8 connection(s)
I20260928 02:53:32.225018 5991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:32.231702 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 5991
I20260928 02:53:32.231962 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:32.232825 28567 external_mini_cluster.cc:1428] Setting key 2f413a15b5ff71c57980fe061333e4a2
I20260928 02:53:32.233983 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.227652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:49589 (local address 127.27.229.254:37853)
0928 02:53:32.228104 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:32.228109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:32.229402 (+ 1293us) server_negotiation.cc:407] Connection header received
0928 02:53:32.231123 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:32.231127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:32.231179 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:32.231278 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:32.233099 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:32.233103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:32.233106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:32.233128 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:32.233169 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:32.233181 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:32.233322 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:32.233586 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:53:32.233660 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260928 02:53:32.235139 6121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.228227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37853 (local address 127.27.229.193:49589)
0928 02:53:32.229188 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:32.229231 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:32.230777 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:32.231573 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:32.231587 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:32.231733 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:32.232692 (+ 959us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:32.232697 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:32.232751 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:32.232899 (+ 148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:32.233292 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:32.233303 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:32.233311 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:53:32.233589 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 02:53:32.233875 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":290,"threads_started":1}
I20260928 02:53:32.235963 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--builtin_ntp_servers=127.27.229.212:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:53:32.236709 6118 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37853
I20260928 02:53:32.237000 6118 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:32.237675 6118 heartbeater.cc:511] Master 127.27.229.254:37853 requested a full tablet report, sending...
I20260928 02:53:32.239897 5932 ts_manager.cc:194] Registered new tserver with Master: 8bdf140f045e4c65ab2652bd8bc75174 (127.27.229.193:37083)
W20260928 02:53:32.564819 6122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:32.565107 6122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:32.565250 6122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:32.572628 6122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:32.572741 6122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:32.572921 6122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:32.583787 6122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:32.585096 6122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:32.586915 6122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:32.600732 6127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:32.600665 6128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:32.600777 6130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:32.602125 6122 server_base.cc:1034] running on GCE node
I20260928 02:53:32.602965 6122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:32.604079 6122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:32.605346 6122 hybrid_clock.cc:648] HybridClock initialized: now 1790564012605278 us; error 66 us; skew 500 ppm
I20260928 02:53:32.605655 6122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:32.608237 6122 webserver.cc:460] Webserver started at http://127.27.229.194:37893/ using document root <none> and password file <none>
I20260928 02:53:32.608996 6122 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:32.609095 6122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:32.609483 6122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:32.612731 6122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "b53a603385d4456083ca8808ba3348b9"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "44714b3256d47189ca27118bd28c0ad2"
server_key_iv: "d6607b86f7a3619569f394ce1f6213ed"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.613631 6122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "b53a603385d4456083ca8808ba3348b9"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "44714b3256d47189ca27118bd28c0ad2"
server_key_iv: "d6607b86f7a3619569f394ce1f6213ed"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.619309 6122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:53:32.623055 6136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.624555 6122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:53:32.624752 6122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "b53a603385d4456083ca8808ba3348b9"
format_stamp: "Formatted at 2026-09-28 02:53:32 on dist-test-slave-s1kg"
server_key: "44714b3256d47189ca27118bd28c0ad2"
server_key_iv: "d6607b86f7a3619569f394ce1f6213ed"
server_key_version: "encryptionkey@0"
I20260928 02:53:32.625025 6122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:32.631580 6122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:32.632543 6122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:32.632853 6122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:32.633977 6122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:32.635617 6122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:32.635749 6122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.635867 6122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:32.635957 6122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:32.672355 6122 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:38975
I20260928 02:53:32.672407 6248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:38975 every 8 connection(s)
I20260928 02:53:32.673733 6122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:32.680787 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6122
I20260928 02:53:32.681010 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:32.681625 28567 external_mini_cluster.cc:1428] Setting key 6e5b61187cfe5ba3e00d3ba1f8a620f8
I20260928 02:53:32.682215 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.676421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:43459 (local address 127.27.229.254:37853)
0928 02:53:32.676615 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:32.676620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:32.678002 (+ 1382us) server_negotiation.cc:407] Connection header received
0928 02:53:32.679704 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:32.679710 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:32.679776 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:32.679873 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:32.681261 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:32.681269 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:32.681274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:32.681296 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:32.681341 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:32.681358 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:32.681529 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:32.681806 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:53:32.681876 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:53:32.683367 6251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:32.676899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37853 (local address 127.27.229.194:43459)
0928 02:53:32.677835 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:32.677875 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:32.679430 (+ 1555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:32.680063 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:32.680071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:32.680158 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:32.680914 (+ 756us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:32.680919 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:32.680971 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:32.681076 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:32.681564 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:32.681573 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:32.681579 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:53:32.681888 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:53:32.682171 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":285,"threads_started":1}
I20260928 02:53:32.684628 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--builtin_ntp_servers=127.27.229.212:46061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 02:53:32.684790 6249 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37853
I20260928 02:53:32.685089 6249 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:32.685668 6249 heartbeater.cc:511] Master 127.27.229.254:37853 requested a full tablet report, sending...
I20260928 02:53:32.687121 5932 ts_manager.cc:194] Registered new tserver with Master: b53a603385d4456083ca8808ba3348b9 (127.27.229.194:38975)
W20260928 02:53:33.010821 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:33.011228 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:33.011371 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:33.019033 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:33.019155 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:33.019330 6252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:33.029114 6252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37853
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:33.030494 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:33.032356 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:33.046099 6257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:33.046242 6260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:33.046268 6258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:33.047070 6252 server_base.cc:1034] running on GCE node
I20260928 02:53:33.047701 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:33.048648 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:33.049862 6252 hybrid_clock.cc:648] HybridClock initialized: now 1790564013049793 us; error 55 us; skew 500 ppm
I20260928 02:53:33.050256 6252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:53:33.052762 6252 webserver.cc:460] Webserver started at http://127.27.229.195:43999/ using document root <none> and password file <none>
I20260928 02:53:33.053481 6252 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:33.053622 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:33.054005 6252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:33.057219 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "c444a08d41024f189019769d8a92b94d"
format_stamp: "Formatted at 2026-09-28 02:53:33 on dist-test-slave-s1kg"
server_key: "5fb460c0cc962ad4ed6a27ab37091e7c"
server_key_iv: "9e804098c7dbd7bbbbfee90571aa92ac"
server_key_version: "encryptionkey@0"
I20260928 02:53:33.058099 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "c444a08d41024f189019769d8a92b94d"
format_stamp: "Formatted at 2026-09-28 02:53:33 on dist-test-slave-s1kg"
server_key: "5fb460c0cc962ad4ed6a27ab37091e7c"
server_key_iv: "9e804098c7dbd7bbbbfee90571aa92ac"
server_key_version: "encryptionkey@0"
I20260928 02:53:33.063585 6252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:53:33.067348 6266 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:33.068768 6252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:53:33.068946 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "c444a08d41024f189019769d8a92b94d"
format_stamp: "Formatted at 2026-09-28 02:53:33 on dist-test-slave-s1kg"
server_key: "5fb460c0cc962ad4ed6a27ab37091e7c"
server_key_iv: "9e804098c7dbd7bbbbfee90571aa92ac"
server_key_version: "encryptionkey@0"
I20260928 02:53:33.069182 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:33.075776 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:33.076659 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:33.076948 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:33.077967 6252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:33.079514 6252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:33.079612 6252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:33.079770 6252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:33.079855 6252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:33.113963 6252 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:39125
I20260928 02:53:33.114076 6378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:39125 every 8 connection(s)
I20260928 02:53:33.115384 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:33.117758 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6252
I20260928 02:53:33.117955 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:33.118456 28567 external_mini_cluster.cc:1428] Setting key 759e4aeae6bc00fec7400d811d233456
I20260928 02:53:33.125087 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.118069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46241 (local address 127.27.229.254:37853)
0928 02:53:33.118287 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:33.118293 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:33.120145 (+ 1852us) server_negotiation.cc:407] Connection header received
0928 02:53:33.122318 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:33.122324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:33.122390 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:33.122507 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:33.124248 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:33.124254 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:33.124259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:33.124289 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:33.124322 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:33.124335 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:33.124487 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:33.124707 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 02:53:33.124780 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:53:33.126077 6381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.118778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37853 (local address 127.27.229.195:46241)
0928 02:53:33.119991 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:33.120039 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:33.122050 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:33.122737 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:33.122746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:33.123176 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:33.123897 (+ 721us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:53:33.123900 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:33.123945 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:53:33.124045 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:33.124465 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:33.124475 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:33.124483 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 02:53:33.124798 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 02:53:33.125077 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":333,"threads_started":1}
I20260928 02:53:33.127519 6379 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37853
I20260928 02:53:33.127849 6379 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:33.128515 6379 heartbeater.cc:511] Master 127.27.229.254:37853 requested a full tablet report, sending...
I20260928 02:53:33.130084 5932 ts_manager.cc:194] Registered new tserver with Master: c444a08d41024f189019769d8a92b94d (127.27.229.195:39125)
I20260928 02:53:33.134161 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 02:53:33.141006 6120 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:53:33.139021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49360 (local address 127.27.229.254:37853)
0928 02:53:33.139190 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:33.139194 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:33.139913 (+ 719us) server_negotiation.cc:407] Connection header received
0928 02:53:33.140147 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:33.140152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:33.140207 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:33.140317 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:33.140660 (+ 343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49360: BlockingRecv error: recv got EOF from 127.0.0.1:49360 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:33.141886 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5991
I20260928 02:53:33.145766 6388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.143020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.27.229.193:37083)
0928 02:53:33.143587 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:33.143601 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:33.143629 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:53:33.143689 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:33.143695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:33.143905 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:33.144044 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:33.144979 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:33.144986 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:33.144997 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:33.145015 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:33.145056 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:33.145071 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:33.145184 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:33.145409 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 02:53:33.145494 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":267,"threads_started":1}
I20260928 02:53:33.147094 6113 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:33.228266 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5991
I20260928 02:53:33.245046 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6122
I20260928 02:53:33.248677 6390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.246030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55994 (local address 127.27.229.194:38975)
0928 02:53:33.246484 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:33.246491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:33.246517 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:53:33.246575 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:33.246581 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:33.246807 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:33.247031 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:33.247962 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:33.247969 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:33.247981 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:33.247992 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:33.248030 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:33.248042 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:33.248192 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:33.248313 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 02:53:33.248420 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":215,"threads_started":1}
I20260928 02:53:33.250034 6244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:33.332180 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6122
I20260928 02:53:33.348381 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6252
I20260928 02:53:33.352210 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:33.349307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60668 (local address 127.27.229.195:39125)
0928 02:53:33.350006 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:33.350014 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:33.350047 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:53:33.350117 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:33.350127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:33.350346 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:33.350477 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:33.351459 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:33.351465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:33.351476 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:53:33.351491 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:33.351529 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:53:33.351553 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:33.351683 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:33.351734 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 02:53:33.351844 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":380,"threads_started":1}
I20260928 02:53:33.353440 6374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:33.438954 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6252
I20260928 02:53:33.455222 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 5901
I20260928 02:53:33.456462 5962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:33.558799 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 5901
2026-09-28T02:53:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2356 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:33 dist-test-slave-s1kg krb5kdc[6397](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:33 dist-test-slave-s1kg krb5kdc[6397](info): set up 2 sockets
Sep 28 02:53:33 dist-test-slave-s1kg krb5kdc[6397](info): commencing operation
krb5kdc: starting...
W20260928 02:53:34.699766 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:53:34 dist-test-slave-s1kg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:53:34.940006 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:40121
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:35.266582 6410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:35.267002 6410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:35.267153 6410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:35.274485 6410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:53:35.274611 6410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:35.274695 6410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:35.274787 6410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:53:35.274907 6410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:53:35.284891 6410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:35.286262 6410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:35.287604 6410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:35.299933 6415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:35.299849 6416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:35.299904 6418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:35.301124 6410 server_base.cc:1034] running on GCE node
I20260928 02:53:35.302042 6410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:35.303570 6410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:35.304792 6410 hybrid_clock.cc:648] HybridClock initialized: now 1790564015304751 us; error 34 us; skew 500 ppm
I20260928 02:53:35.305114 6410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:35 dist-test-slave-s1kg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:35.313200 6410 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:53:35.314941 6410 webserver.cc:460] Webserver started at http://127.27.229.254:44519/ using document root <none> and password file <none>
I20260928 02:53:35.315666 6410 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:35.315797 6410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:35.316103 6410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:35.319182 6410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "4269446caebc4201a359c052a0b5a57f"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "8186cbf81eaa413d0b6a292d090433b6"
server_key_iv: "3e7d311d3bd99c2a2b87dba79e6f2fb1"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.320158 6410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "4269446caebc4201a359c052a0b5a57f"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "8186cbf81eaa413d0b6a292d090433b6"
server_key_iv: "3e7d311d3bd99c2a2b87dba79e6f2fb1"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.325919 6410 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:53:35.329583 6425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:35.331032 6410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:35.331207 6410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "4269446caebc4201a359c052a0b5a57f"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "8186cbf81eaa413d0b6a292d090433b6"
server_key_iv: "3e7d311d3bd99c2a2b87dba79e6f2fb1"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.331393 6410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:35.352303 6410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:35.356455 6410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:35.356755 6410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:35.377326 6410 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:40121
I20260928 02:53:35.377364 6476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:40121 every 8 connection(s)
I20260928 02:53:35.378783 6410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:53:35.382498 6477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:35.383576 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6410
I20260928 02:53:35.383836 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:53:35.384577 28567 external_mini_cluster.cc:1428] Setting key abace1d234806b1721400307232e199c
I20260928 02:53:35.397390 6477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: Bootstrap starting.
Sep 28 02:53:35 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:35.400781 6477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:35.402273 6477 log.cc:824] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:35.405814 6477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: No bootstrap required, opened a new log
I20260928 02:53:35.409809 6480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:35.386784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.27.229.254:40121)
0928 02:53:35.387690 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:35.387705 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:35.387758 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:53:35.389467 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:35.389499 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:35.390080 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:35.390552 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:35.392026 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:35.394080 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:35.395536 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:35.396601 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:35.396804 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:35.400712 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:35.400724 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:35.400746 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:35.400807 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:35.404871 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:35.405643 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:35.405651 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:35.405661 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:35.405878 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:35.406358 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:35.406364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:35.406367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:35.407109 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:35.407418 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:35.407796 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 02:53:35.408286 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":363,"threads_started":1}
I20260928 02:53:35.415453 6477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } }
I20260928 02:53:35.415824 6477 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:35.415897 6477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4269446caebc4201a359c052a0b5a57f, State: Initialized, Role: FOLLOWER
I20260928 02:53:35.416476 6477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } }
I20260928 02:53:35.416680 6477 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:53:35.416780 6477 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:53:35.416896 6477 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:35.419239 6477 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } }
I20260928 02:53:35.419893 6477 leader_election.cc:302] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4269446caebc4201a359c052a0b5a57f; no voters:
I20260928 02:53:35.420482 6477 leader_election.cc:288] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:53:35.420643 6482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:35.421432 6482 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [term 1 LEADER]: Becoming Leader. State: Replica: 4269446caebc4201a359c052a0b5a57f, State: Running, Role: LEADER
I20260928 02:53:35.421967 6482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } }
I20260928 02:53:35.422708 6477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:53:35.424607 6483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4269446caebc4201a359c052a0b5a57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } } }
I20260928 02:53:35.425004 6483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:35.424980 6484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4269446caebc4201a359c052a0b5a57f. Latest consensus state: current_term: 1 leader_uuid: "4269446caebc4201a359c052a0b5a57f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4269446caebc4201a359c052a0b5a57f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40121 } } }
I20260928 02:53:35.425295 6484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f [sys.catalog]: This master's current role is: LEADER
I20260928 02:53:35.428256 6488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:53:35.435549 6488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:53:35.444948 6488 catalog_manager.cc:1347] Generated new cluster ID: 23a6e5dc51144ab7a44be67330083c2f
I20260928 02:53:35.445138 6488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:53:35.466946 6488 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:53:35.468878 6488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:53:35.484687 6488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: Generated new TSK 0
I20260928 02:53:35.485795 6488 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:53:35.547907 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:35.889887 6503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:35.890261 6503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:35.890439 6503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:35.897935 6503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:35.898120 6503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:35.898312 6503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:53:35.908504 6503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:35.910125 6503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:35.911593 6503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:35.925267 6508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:35.925365 6511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:35.925570 6509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:35.926231 6503 server_base.cc:1034] running on GCE node
I20260928 02:53:35.926951 6503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:35.928231 6503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:35.929476 6503 hybrid_clock.cc:648] HybridClock initialized: now 1790564015929411 us; error 64 us; skew 500 ppm
I20260928 02:53:35.929867 6503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:35 dist-test-slave-s1kg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:35.933357 6503 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:53:35.935179 6503 webserver.cc:460] Webserver started at http://127.27.229.193:43721/ using document root <none> and password file <none>
I20260928 02:53:35.935899 6503 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:35.936060 6503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:35.936399 6503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:35.939462 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "49bb6747254141dfbc6d13baf1fcd2b5"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "28a22a0ba9402d9fb81481f597dda0f6"
server_key_iv: "31951f233179d166ff6e04dcca3141b3"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.940296 6503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "49bb6747254141dfbc6d13baf1fcd2b5"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "28a22a0ba9402d9fb81481f597dda0f6"
server_key_iv: "31951f233179d166ff6e04dcca3141b3"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.945827 6503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:53:35.949674 6518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:35.951251 6503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:53:35.951442 6503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "49bb6747254141dfbc6d13baf1fcd2b5"
format_stamp: "Formatted at 2026-09-28 02:53:35 on dist-test-slave-s1kg"
server_key: "28a22a0ba9402d9fb81481f597dda0f6"
server_key_iv: "31951f233179d166ff6e04dcca3141b3"
server_key_version: "encryptionkey@0"
I20260928 02:53:35.951650 6503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:35.975615 6503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:35.980266 6503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:35.980603 6503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:35.981666 6503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:35.983142 6503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:35.983256 6503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:35.983409 6503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:35.983510 6503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:36.019399 6503 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37473
I20260928 02:53:36.019464 6630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37473 every 8 connection(s)
I20260928 02:53:36.020712 6503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:53:36.024346 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6503
I20260928 02:53:36.024608 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:53:36.025374 28567 external_mini_cluster.cc:1428] Setting key 02880021836a07b5923eabdfbdf78adc
Sep 28 02:53:36 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564015, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:36.041692 6633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.023404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:37457 (local address 127.27.229.254:40121)
0928 02:53:36.023973 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:36.023979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:36.025121 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 02:53:36.027508 (+ 2387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:36.027514 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:36.027591 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:36.027718 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:36.029263 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.029981 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.031721 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.032121 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.032259 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:36.035852 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:36.035857 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:36.035860 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:36.035890 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:36.038550 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:36.039377 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:36.039386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:36.039388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:36.039502 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:36.039928 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:36.039933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:36.039936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:36.040194 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:36.040345 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:36.041067 (+ 722us) server_negotiation.cc:299] Negotiation successful
0928 02:53:36.041283 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":334,"threads_started":1}
I20260928 02:53:36.042670 6634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.023978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40121 (local address 127.27.229.193:37457)
0928 02:53:36.024943 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:36.024984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:36.027205 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:36.027957 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:36.027966 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:36.028488 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:36.029106 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:36.029124 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.030179 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.030183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:36.031571 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:36.031583 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.032350 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.032358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:36.032588 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:36.033312 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:36.033334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:36.035642 (+ 2308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:36.038717 (+ 3075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:36.038726 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:36.038744 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:36.039245 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:36.039619 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:36.039625 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:36.039628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:36.039810 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:36.040319 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:36.040327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:36.040689 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 02:53:36.040951 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:53:36.041503 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":322,"threads_started":1}
I20260928 02:53:36.044360 6631 heartbeater.cc:347] Connected to a master server at 127.27.229.254:40121
I20260928 02:53:36.044713 6631 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:36.045428 6631 heartbeater.cc:511] Master 127.27.229.254:40121 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:53:36.047694 6442 ts_manager.cc:194] Registered new tserver with Master: 49bb6747254141dfbc6d13baf1fcd2b5 (127.27.229.193:37473)
I20260928 02:53:36.049126 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:37457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:53:36.078166 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:36.416550 6637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:36.416869 6637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:36.417040 6637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:36.424973 6637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:36.425115 6637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:36.425298 6637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:53:36.435590 6637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:36.437011 6637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:36.438339 6637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:36.453174 6642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:36.453255 6643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:36.453215 6645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:36.454128 6637 server_base.cc:1034] running on GCE node
I20260928 02:53:36.454774 6637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:36.455875 6637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:36.457125 6637 hybrid_clock.cc:648] HybridClock initialized: now 1790564016457075 us; error 38 us; skew 500 ppm
I20260928 02:53:36.457445 6637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:36 dist-test-slave-s1kg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:36.461625 6637 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:53:36.463383 6637 webserver.cc:460] Webserver started at http://127.27.229.194:34143/ using document root <none> and password file <none>
I20260928 02:53:36.464165 6637 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:36.464308 6637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:36.464655 6637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:36.467912 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "aed6c8ff035f422fb269ba7f1d7870dc"
format_stamp: "Formatted at 2026-09-28 02:53:36 on dist-test-slave-s1kg"
server_key: "8078e33f511b6f1d27f277081dfe3479"
server_key_iv: "c855a5e49b7a2dfdd2bda4dfa2364125"
server_key_version: "encryptionkey@0"
I20260928 02:53:36.468792 6637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "aed6c8ff035f422fb269ba7f1d7870dc"
format_stamp: "Formatted at 2026-09-28 02:53:36 on dist-test-slave-s1kg"
server_key: "8078e33f511b6f1d27f277081dfe3479"
server_key_iv: "c855a5e49b7a2dfdd2bda4dfa2364125"
server_key_version: "encryptionkey@0"
I20260928 02:53:36.474238 6637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:53:36.477953 6652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:36.479624 6637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:53:36.479821 6637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "aed6c8ff035f422fb269ba7f1d7870dc"
format_stamp: "Formatted at 2026-09-28 02:53:36 on dist-test-slave-s1kg"
server_key: "8078e33f511b6f1d27f277081dfe3479"
server_key_iv: "c855a5e49b7a2dfdd2bda4dfa2364125"
server_key_version: "encryptionkey@0"
I20260928 02:53:36.480041 6637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:36.499796 6637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:36.504829 6637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:36.505177 6637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:36.506278 6637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:36.507876 6637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:36.507992 6637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:36.508169 6637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:36.508251 6637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:36.543248 6637 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:45089
I20260928 02:53:36.543293 6764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:45089 every 8 connection(s)
I20260928 02:53:36.544528 6637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:53:36.551923 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6637
I20260928 02:53:36.552111 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:53:36.552541 28567 external_mini_cluster.cc:1428] Setting key aa52c9157b3145370dd85d2237d41e53
Sep 28 02:53:36 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:36.565227 6767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.547040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:55361 (local address 127.27.229.254:40121)
0928 02:53:36.547557 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:36.547562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:36.548443 (+ 881us) server_negotiation.cc:407] Connection header received
0928 02:53:36.550198 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:36.550204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:36.550294 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:36.550413 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:36.551951 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.552824 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.554634 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.555114 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.555245 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:36.559095 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:36.559100 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:36.559104 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:36.559136 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:36.561885 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:36.562722 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:36.562736 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:36.562740 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:36.562938 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:36.563408 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:36.563412 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:36.563416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:36.563708 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:36.563880 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:36.564635 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 02:53:36.564859 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":266,"threads_started":1}
I20260928 02:53:36.566252 6768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:36.547632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40121 (local address 127.27.229.194:55361)
0928 02:53:36.548286 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:36.548327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:36.549941 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:36.550638 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:36.550645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:36.551202 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:36.551787 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:36.551804 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.552978 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.552982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:36.554491 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:36.554503 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:36.555336 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:36.555343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:36.555624 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:36.556296 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:36.556323 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:36.558776 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:36.562062 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:36.562072 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:36.562091 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:36.562583 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:36.563071 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:36.563076 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:36.563080 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:36.563288 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:36.563847 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:36.563856 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:36.564224 (+ 368us) client_negotiation.cc:769] Sending connection context
0928 02:53:36.564530 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 02:53:36.565082 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":250,"threads_started":1}
I20260928 02:53:36.567790 6765 heartbeater.cc:347] Connected to a master server at 127.27.229.254:40121
I20260928 02:53:36.568125 6765 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:36.568997 6765 heartbeater.cc:511] Master 127.27.229.254:40121 requested a full tablet report, sending...
I20260928 02:53:36.570533 6442 ts_manager.cc:194] Registered new tserver with Master: aed6c8ff035f422fb269ba7f1d7870dc (127.27.229.194:45089)
I20260928 02:53:36.571427 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:55361
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:53:36.607407 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:53:36.958774 6771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:53:36.959283 6771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:53:36.959522 6771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:53:36.967412 6771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:53:36.967592 6771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:53:36.967711 6771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:53:36.977439 6771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:40121
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:53:36.978703 6771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:53:36.980113 6771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:53:36.993484 6777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:36.993670 6779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:53:36.993856 6776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:53:36.994329 6771 server_base.cc:1034] running on GCE node
I20260928 02:53:36.995055 6771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:53:36.996037 6771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:53:36.997288 6771 hybrid_clock.cc:648] HybridClock initialized: now 1790564016997245 us; error 40 us; skew 500 ppm
I20260928 02:53:36.997725 6771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:53:36 dist-test-slave-s1kg krb5kdc[6397](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:53:37.001153 6771 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:53:37.002771 6771 webserver.cc:460] Webserver started at http://127.27.229.195:35063/ using document root <none> and password file <none>
I20260928 02:53:37.003549 6771 fs_manager.cc:359] Metadata directory not provided
I20260928 02:53:37.003708 6771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:53:37.004020 6771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:53:37.007146 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "c0d7cc36e8064cd2b128a20a9b547f62"
format_stamp: "Formatted at 2026-09-28 02:53:37 on dist-test-slave-s1kg"
server_key: "51293067ff1e950c59e6fd85e4600c98"
server_key_iv: "df3a02e467bc900e62f426eb4d9d9ab6"
server_key_version: "encryptionkey@0"
I20260928 02:53:37.007948 6771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "c0d7cc36e8064cd2b128a20a9b547f62"
format_stamp: "Formatted at 2026-09-28 02:53:37 on dist-test-slave-s1kg"
server_key: "51293067ff1e950c59e6fd85e4600c98"
server_key_iv: "df3a02e467bc900e62f426eb4d9d9ab6"
server_key_version: "encryptionkey@0"
I20260928 02:53:37.013257 6771 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:53:37.016909 6786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:37.018294 6771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:53:37.018479 6771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "c0d7cc36e8064cd2b128a20a9b547f62"
format_stamp: "Formatted at 2026-09-28 02:53:37 on dist-test-slave-s1kg"
server_key: "51293067ff1e950c59e6fd85e4600c98"
server_key_iv: "df3a02e467bc900e62f426eb4d9d9ab6"
server_key_version: "encryptionkey@0"
I20260928 02:53:37.018692 6771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:53:37.037865 6771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:53:37.042399 6771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:53:37.042721 6771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:53:37.043730 6771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:53:37.045133 6771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:53:37.045236 6771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:37.045385 6771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:53:37.045456 6771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:53:37.052356 6631 heartbeater.cc:503] Master 127.27.229.254:40121 was elected leader, sending a full tablet report...
I20260928 02:53:37.080106 6771 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:39515
I20260928 02:53:37.080190 6898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:39515 every 8 connection(s)
I20260928 02:53:37.081374 6771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:53:37.082254 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 6771
I20260928 02:53:37.082458 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:53:37.082976 28567 external_mini_cluster.cc:1428] Setting key 7b031a4dd534bf2673ccd7afce4a26b2
Sep 28 02:53:37 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564016, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:53:37.103293 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.084276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:55001 (local address 127.27.229.254:40121)
0928 02:53:37.084786 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.084791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.086005 (+ 1214us) server_negotiation.cc:407] Connection header received
0928 02:53:37.088157 (+ 2152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.088161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.088223 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.088311 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:37.090001 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.090764 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.092373 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.092776 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.092874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.096442 (+ 3568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:37.096447 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:37.096450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:37.096525 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:37.100197 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.100961 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.100966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.100968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.101085 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.101622 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.101627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.101629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.101909 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:37.102073 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.102748 (+ 675us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.103021 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":250,"threads_started":1}
I20260928 02:53:37.104269 6902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.084924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40121 (local address 127.27.229.195:55001)
0928 02:53:37.085834 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.085884 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.087904 (+ 2020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.088591 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.088599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.089181 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:53:37.089822 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.089840 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.090961 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.090966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.092226 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.092238 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.092979 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.092984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.093191 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.093856 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:53:37.093882 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:53:37.096224 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:53:37.100335 (+ 4111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:37.100342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:37.100357 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:37.100830 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:37.101275 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:53:37.101279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:53:37.101282 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:53:37.101486 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:53:37.102029 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:53:37.102035 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:53:37.102382 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.102646 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.103161 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":351,"threads_started":1}
I20260928 02:53:37.105738 6899 heartbeater.cc:347] Connected to a master server at 127.27.229.254:40121
I20260928 02:53:37.106036 6899 heartbeater.cc:464] Registering TS with master...
I20260928 02:53:37.106799 6899 heartbeater.cc:511] Master 127.27.229.254:40121 requested a full tablet report, sending...
I20260928 02:53:37.108481 6442 ts_manager.cc:194] Registered new tserver with Master: c0d7cc36e8064cd2b128a20a9b547f62 (127.27.229.195:39515)
I20260928 02:53:37.109314 6442 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:55001
I20260928 02:53:37.112033 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:53:37.129001 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.116601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.27.229.254:40121)
0928 02:53:37.116845 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.116854 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.117248 (+ 394us) server_negotiation.cc:407] Connection header received
0928 02:53:37.117369 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.117373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.117446 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.117566 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:37.118988 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.119703 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.120971 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.121483 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.121604 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.124195 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:37.124200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:37.124203 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:37.124233 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:37.126683 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.127391 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.127395 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.127397 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.127515 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.127868 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.127871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.127873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.128177 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:37.128351 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.128633 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.128790 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:53:37.133124 6442 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:53:37.140229 6442 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:53:37.187975 6916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.153860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:39515 (local address 127.0.0.1:60480)
0928 02:53:37.178610 (+ 24750us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.178637 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.178789 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.179762 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.179770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.179793 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:37.180265 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.180278 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.182254 (+ 1976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.182257 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.184729 (+ 2472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.184743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.186861 (+ 2118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.186868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.187083 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.187137 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.187260 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.187497 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24573,"mutex_wait_us":174,"thread_start_us":24174,"threads_started":1}
I20260928 02:53:37.188127 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.153796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58478 (local address 127.27.229.193:37473)
0928 02:53:37.179745 (+ 25949us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.179753 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.179781 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:53:37.179880 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.179888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.180131 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.180344 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:37.180879 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.181829 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.185294 (+ 3465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.186651 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.187398 (+ 747us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.187561 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.187638 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.187790 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25800,"thread_start_us":25650,"threads_started":1}
I20260928 02:53:37.188112 6912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.153288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:45089 (local address 127.0.0.1:36788)
0928 02:53:37.164800 (+ 11512us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.164852 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.165073 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.179390 (+ 14317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.179399 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.179442 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:37.179980 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.179997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.182143 (+ 2146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.182147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.185111 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.185125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.187161 (+ 2036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.187170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.187396 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.187441 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.187534 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.187676 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11180,"mutex_wait_us":131,"thread_start_us":11012,"threads_started":1}
I20260928 02:53:37.188832 6913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.153286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37473 (local address 127.0.0.1:58478)
0928 02:53:37.177941 (+ 24655us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.177967 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.178136 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.180323 (+ 2187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.180328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.180352 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:37.180690 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.180698 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.181965 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.181973 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.185111 (+ 3138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.185125 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.186970 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.186976 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.187112 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.187139 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.187260 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.188592 (+ 1332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24482,"mutex_wait_us":1544,"thread_start_us":24109,"threads_started":1}
I20260928 02:53:37.189199 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.153431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36788 (local address 127.27.229.194:45089)
0928 02:53:37.178656 (+ 25225us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.178664 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.178694 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:53:37.178803 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.178812 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.179126 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.180115 (+ 989us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:37.180328 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.181959 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.185269 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.186893 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.188400 (+ 1507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.188599 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.188675 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.188845 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25059,"thread_start_us":25490,"threads_started":1}
I20260928 02:53:37.189199 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.154062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.27.229.195:39515)
0928 02:53:37.177148 (+ 23086us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.177162 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.179216 (+ 2054us) server_negotiation.cc:407] Connection header received
0928 02:53:37.179340 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.179350 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.179573 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.179764 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:37.180483 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.182106 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.184952 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.186619 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.186871 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.187967 (+ 1096us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.188053 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.188685 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22819,"thread_start_us":23105,"threads_started":1}
I20260928 02:53:37.192909 6834 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe590cd14544e290a467f442d56a6a (DEFAULT_TABLE table=test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:37.192826 6566 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe590cd14544e290a467f442d56a6a (DEFAULT_TABLE table=test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:37.192822 6700 tablet_service.cc:1467] Processing CreateTablet for tablet 3efe590cd14544e290a467f442d56a6a (DEFAULT_TABLE table=test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:53:37.194149 6834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe590cd14544e290a467f442d56a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:37.194177 6700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe590cd14544e290a467f442d56a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:37.195266 6566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3efe590cd14544e290a467f442d56a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:53:37.205876 6918 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Bootstrap starting.
I20260928 02:53:37.208674 6918 tablet_bootstrap.cc:654] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:37.209774 6918 log.cc:824] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:37.210119 6919 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Bootstrap starting.
I20260928 02:53:37.211297 6920 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Bootstrap starting.
I20260928 02:53:37.214030 6919 tablet_bootstrap.cc:654] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:37.214967 6920 tablet_bootstrap.cc:654] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Neither blocks nor log segments found. Creating new log.
I20260928 02:53:37.215268 6919 log.cc:824] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:37.215363 6918 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: No bootstrap required, opened a new log
I20260928 02:53:37.215613 6918 ts_tablet_manager.cc:1397] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260928 02:53:37.216085 6920 log.cc:824] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:53:37.219245 6920 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: No bootstrap required, opened a new log
I20260928 02:53:37.219533 6920 ts_tablet_manager.cc:1397] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260928 02:53:37.220155 6919 tablet_bootstrap.cc:492] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: No bootstrap required, opened a new log
I20260928 02:53:37.220399 6919 ts_tablet_manager.cc:1397] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260928 02:53:37.226239 6918 raft_consensus.cc:348] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.226682 6918 raft_consensus.cc:374] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:37.226812 6918 raft_consensus.cc:729] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed6c8ff035f422fb269ba7f1d7870dc, State: Initialized, Role: FOLLOWER
I20260928 02:53:37.227453 6918 consensus_queue.cc:260] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.227835 6920 raft_consensus.cc:348] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.228338 6920 raft_consensus.cc:374] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:37.228499 6920 raft_consensus.cc:729] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bb6747254141dfbc6d13baf1fcd2b5, State: Initialized, Role: FOLLOWER
I20260928 02:53:37.228632 6765 heartbeater.cc:503] Master 127.27.229.254:40121 was elected leader, sending a full tablet report...
I20260928 02:53:37.229213 6918 ts_tablet_manager.cc:1428] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260928 02:53:37.229102 6920 consensus_queue.cc:260] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.231564 6919 raft_consensus.cc:348] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.231966 6919 raft_consensus.cc:374] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:53:37.232079 6919 raft_consensus.cc:729] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0d7cc36e8064cd2b128a20a9b547f62, State: Initialized, Role: FOLLOWER
I20260928 02:53:37.232205 6920 ts_tablet_manager.cc:1428] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260928 02:53:37.232657 6919 consensus_queue.cc:260] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.234853 6899 heartbeater.cc:503] Master 127.27.229.254:40121 was elected leader, sending a full tablet report...
I20260928 02:53:37.234804 6924 raft_consensus.cc:482] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:53:37.235138 6924 raft_consensus.cc:504] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.235405 6919 ts_tablet_manager.cc:1428] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 02:53:37.237147 6924 leader_election.cc:288] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49bb6747254141dfbc6d13baf1fcd2b5 (127.27.229.193:37473), c0d7cc36e8064cd2b128a20a9b547f62 (127.27.229.195:39515)
I20260928 02:53:37.245031 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.237889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:59495 (local address 127.27.229.195:39515)
0928 02:53:37.238130 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.238137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.238449 (+ 312us) server_negotiation.cc:407] Connection header received
0928 02:53:37.238738 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.238742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.238808 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.238897 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:37.239613 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.240734 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.243040 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.244125 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.244313 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.244387 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.244670 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.244790 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:53:37.245193 6928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.237437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:39515 (local address 127.27.229.194:59495)
0928 02:53:37.238360 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.238378 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.238560 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.239088 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.239091 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.239116 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:37.239462 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.239471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.240918 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.240928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.242864 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.242880 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.244337 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.244343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.244513 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.244535 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.244638 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.244776 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":94,"thread_start_us":607,"threads_started":1}
I20260928 02:53:37.245492 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.237616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:42809 (local address 127.27.229.193:37473)
0928 02:53:37.237804 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.237810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.238459 (+ 649us) server_negotiation.cc:407] Connection header received
0928 02:53:37.238690 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.238699 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.238769 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.238883 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:53:37.239674 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.240732 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.243573 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.244437 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.244554 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.244602 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.245097 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.245231 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 02:53:37.245636 6927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.237260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37473 (local address 127.27.229.194:42809)
0928 02:53:37.238347 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0928 02:53:37.238368 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:53:37.238531 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:53:37.238933 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:53:37.238938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:53:37.238965 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:53:37.239465 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.239493 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.240866 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.240870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.243316 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:53:37.243337 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.244770 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.244774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:53:37.244909 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.244931 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:53:37.245070 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 02:53:37.245216 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":918,"mutex_wait_us":50,"thread_start_us":551,"threads_started":1}
I20260928 02:53:37.246186 6854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe590cd14544e290a467f442d56a6a" candidate_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" is_pre_election: true
I20260928 02:53:37.246670 6586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe590cd14544e290a467f442d56a6a" candidate_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" is_pre_election: true
I20260928 02:53:37.246805 6854 raft_consensus.cc:2454] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed6c8ff035f422fb269ba7f1d7870dc in term 0.
I20260928 02:53:37.247275 6586 raft_consensus.cc:2454] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aed6c8ff035f422fb269ba7f1d7870dc in term 0.
I20260928 02:53:37.247642 6656 leader_election.cc:302] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aed6c8ff035f422fb269ba7f1d7870dc, c0d7cc36e8064cd2b128a20a9b547f62; no voters:
I20260928 02:53:37.248080 6924 raft_consensus.cc:2781] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:53:37.248229 6924 raft_consensus.cc:482] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:53:37.248368 6924 raft_consensus.cc:3037] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:37.250890 6924 raft_consensus.cc:504] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.251595 6924 leader_election.cc:288] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [CANDIDATE]: Term 1 election: Requested vote from peers 49bb6747254141dfbc6d13baf1fcd2b5 (127.27.229.193:37473), c0d7cc36e8064cd2b128a20a9b547f62 (127.27.229.195:39515)
I20260928 02:53:37.252039 6586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe590cd14544e290a467f442d56a6a" candidate_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49bb6747254141dfbc6d13baf1fcd2b5"
I20260928 02:53:37.252205 6854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3efe590cd14544e290a467f442d56a6a" candidate_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0d7cc36e8064cd2b128a20a9b547f62"
I20260928 02:53:37.252269 6586 raft_consensus.cc:3037] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:37.252424 6854 raft_consensus.cc:3037] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:53:37.255870 6854 raft_consensus.cc:2454] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed6c8ff035f422fb269ba7f1d7870dc in term 1.
I20260928 02:53:37.255870 6586 raft_consensus.cc:2454] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aed6c8ff035f422fb269ba7f1d7870dc in term 1.
I20260928 02:53:37.256309 6655 leader_election.cc:302] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49bb6747254141dfbc6d13baf1fcd2b5, aed6c8ff035f422fb269ba7f1d7870dc; no voters:
I20260928 02:53:37.256688 6924 raft_consensus.cc:2781] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:53:37.257236 6924 raft_consensus.cc:686] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 1 LEADER]: Becoming Leader. State: Replica: aed6c8ff035f422fb269ba7f1d7870dc, State: Running, Role: LEADER
I20260928 02:53:37.257740 6924 consensus_queue.cc:237] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } }
I20260928 02:53:37.262624 6441 catalog_manager.cc:5495] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc reported cstate change: term changed from 0 to 1, leader changed from <none> to aed6c8ff035f422fb269ba7f1d7870dc (127.27.229.194). New cstate: current_term: 1 leader_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aed6c8ff035f422fb269ba7f1d7870dc" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45089 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 } health_report { overall_health: UNKNOWN } } }
W20260928 02:53:37.273170 6632 tablet.cc:2369] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:37.293892 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.288459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36802 (local address 127.27.229.194:45089)
0928 02:53:37.288675 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.288681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.288726 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:53:37.288980 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.288984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.289079 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.289245 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:53:37.289610 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.290346 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.291782 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.292182 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.292318 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.292808 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:53:37.293306 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:53:37.293464 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.293540 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.293619 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 02:53:37.296478 6766 tablet.cc:2369] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:53:37.302636 6854 raft_consensus.cc:1261] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 1 FOLLOWER]: Refusing update from remote peer aed6c8ff035f422fb269ba7f1d7870dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:37.302690 6586 raft_consensus.cc:1261] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 1 FOLLOWER]: Refusing update from remote peer aed6c8ff035f422fb269ba7f1d7870dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:53:37.303721 6929 consensus_queue.cc:1035] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bb6747254141dfbc6d13baf1fcd2b5" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37473 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:37.304068 6924 consensus_queue.cc:1035] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0d7cc36e8064cd2b128a20a9b547f62" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:53:37.312673 6932 mvcc.cc:205] Tried to move back new op lower bound from 7334150214861606912 to 7334150214692712448. Current Snapshot: MvccSnapshot[applied={T|T < 7334150214861606912}]
I20260928 02:53:37.329288 6442 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:37.329581 6442 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:53:37.333304 6442 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4269446caebc4201a359c052a0b5a57f: Sending DeleteTablet for 3 replicas of tablet 3efe590cd14544e290a467f442d56a6a
I20260928 02:53:37.334688 6566 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efe590cd14544e290a467f442d56a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:58478
I20260928 02:53:37.335206 6834 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efe590cd14544e290a467f442d56a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60480
I20260928 02:53:37.336338 6942 tablet_replica.cc:330] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: stopping tablet replica
I20260928 02:53:37.336328 6700 tablet_service.cc:1514] Processing DeleteTablet for tablet 3efe590cd14544e290a467f442d56a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:53:37 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:36788
I20260928 02:53:37.336755 6942 raft_consensus.cc:2229] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:37.337307 6942 raft_consensus.cc:2258] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:37.337376 6944 tablet_replica.cc:330] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: stopping tablet replica
I20260928 02:53:37.338253 6943 tablet_replica.cc:330] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: stopping tablet replica
I20260928 02:53:37.338608 6943 raft_consensus.cc:2229] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:53:37.338922 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6503
I20260928 02:53:37.339075 6943 raft_consensus.cc:2258] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:37.340056 6942 ts_tablet_manager.cc:1905] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:37.340935 6944 raft_consensus.cc:2229] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:53:37.342761 6943 ts_tablet_manager.cc:1905] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:37.343454 6944 raft_consensus.cc:2258] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:53:37.348887 6944 ts_tablet_manager.cc:1905] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:53:37.350153 6942 ts_tablet_manager.cc:1918] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:37.350466 6942 log.cc:1196] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/3efe590cd14544e290a467f442d56a6a
I20260928 02:53:37.350773 6943 ts_tablet_manager.cc:1918] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:37.350982 6943 log.cc:1196] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/3efe590cd14544e290a467f442d56a6a
I20260928 02:53:37.351580 6943 ts_tablet_manager.cc:1939] T 3efe590cd14544e290a467f442d56a6a P c0d7cc36e8064cd2b128a20a9b547f62: Deleting consensus metadata
I20260928 02:53:37.352475 6942 ts_tablet_manager.cc:1939] T 3efe590cd14544e290a467f442d56a6a P 49bb6747254141dfbc6d13baf1fcd2b5: Deleting consensus metadata
I20260928 02:53:37.353540 6429 catalog_manager.cc:4841] TS c0d7cc36e8064cd2b128a20a9b547f62 (127.27.229.195:39515): tablet 3efe590cd14544e290a467f442d56a6a (table test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]) successfully deleted
I20260928 02:53:37.354379 6428 catalog_manager.cc:4841] TS 49bb6747254141dfbc6d13baf1fcd2b5 (127.27.229.193:37473): tablet 3efe590cd14544e290a467f442d56a6a (table test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]) successfully deleted
I20260928 02:53:37.358745 6944 ts_tablet_manager.cc:1918] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:53:37.359019 6944 log.cc:1196] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/3efe590cd14544e290a467f442d56a6a
I20260928 02:53:37.359668 6944 ts_tablet_manager.cc:1939] T 3efe590cd14544e290a467f442d56a6a P aed6c8ff035f422fb269ba7f1d7870dc: Deleting consensus metadata
I20260928 02:53:37.362963 6426 catalog_manager.cc:4841] TS aed6c8ff035f422fb269ba7f1d7870dc (127.27.229.194:45089): tablet 3efe590cd14544e290a467f442d56a6a (table test-table [id=1e62cb7fb8fd43039c1f3cfaca5fa3b7]) successfully deleted
Sep 28 02:53:37 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:53:37.374805 6915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.340447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58484 (local address 127.27.229.193:37473)
0928 02:53:37.341004 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.341010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.355235 (+ 14225us) server_negotiation.cc:407] Connection header received
0928 02:53:37.355437 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.355442 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.355533 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.355643 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:37.357330 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.358447 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.360872 (+ 2425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.361523 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.361656 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.368041 (+ 6385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:37.368057 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:37.368074 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:37.368120 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:37.371688 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.372502 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.372510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.372513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.372668 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.373112 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.373118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.373121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.373502 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:37.373838 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.374166 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.374404 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418}
I20260928 02:53:37.375627 6626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:37.507982 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6503
I20260928 02:53:37.531402 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6637
Sep 28 02:53:37 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:53:37.552008 6914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.532239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36812 (local address 127.27.229.194:45089)
0928 02:53:37.532411 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.532416 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.532433 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:53:37.532649 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.532654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.532748 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.532841 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:37.534598 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.535659 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.537712 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.538170 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.538286 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.544159 (+ 5873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:37.544174 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:37.544198 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:37.544262 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:37.548580 (+ 4318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.549534 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.549544 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.549548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.549756 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.550205 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.550211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.550217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.550701 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:37.551065 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.551350 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.551566 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:53:37.552745 6760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:37.668498 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6637
I20260928 02:53:37.687108 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6771
Sep 28 02:53:37 dist-test-slave-s1kg krb5kdc[6397](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:53:37.703821 6917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:53:37.687967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60484 (local address 127.27.229.195:39515)
0928 02:53:37.688162 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:53:37.688169 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:53:37.688221 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:53:37.688477 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:53:37.688482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:53:37.688559 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:53:37.688689 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:53:37.690102 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.690970 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.692367 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:53:37.692816 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:53:37.692964 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:53:37.697874 (+ 4910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:53:37.697890 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:53:37.697906 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:53:37.697945 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:53:37.700820 (+ 2875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.701652 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.701660 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.701662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.701820 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:53:37.702221 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:53:37.702227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:53:37.702230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:53:37.702588 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:53:37.703072 (+ 484us) server_negotiation.cc:1035] Waiting for connection context
0928 02:53:37.703245 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:53:37.703466 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:53:37.704526 6894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:37.793015 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6771
I20260928 02:53:37.810778 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 6410
I20260928 02:53:37.812100 6472 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:53:37.920248 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 6410
2026-09-28T02:53:37Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4363 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:53:38 dist-test-slave-s1kg krb5kdc[6952](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:53:38 dist-test-slave-s1kg krb5kdc[6952](info): set up 2 sockets
Sep 28 02:53:38 dist-test-slave-s1kg krb5kdc[6952](info): commencing operation
krb5kdc: starting...
W20260928 02:53:39.056085 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:53:39 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:53:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:53:39Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangeradmin_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangeradmin_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangerlookup_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangerlookup_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
I20260928 02:53:39.406760 28567 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhPicry/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/postgres -l logfile start
2026-09-28 02:53:42.241 UTC [6982] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:53:42.241 UTC [6982] LOG: listening on IPv4 address "127.27.229.212", port 53067
2026-09-28 02:53:42.245 UTC [6982] LOG: listening on Unix socket "/tmp/.s.PGSQL.53067"
2026-09-28 02:53:42.268 UTC [6985] LOG: database system was shut down at 2026-09-28 02:53:40 UTC
2026-09-28 02:53:42.272 UTC [6982] LOG: database system is ready to accept connections
I20260928 02:53:43.268107 28567 mini_postgres.cc:96] Postgres bound to 53067
2026-09-28 02:53:43.287 UTC [6993] FATAL: database "slave" does not exist
127.27.229.212:53067 - accepting connections
I20260928 02:53:43.288556 28567 mini_ranger.cc:161] Starting Ranger...
I20260928 02:53:43.316727 28567 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:53:43.869712 28567 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:53:43.869860 28567 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin
2026-09-28 02:53:44,173 [I] DB FLAVOR :POSTGRES
2026-09-28 02:53:44,174 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:53:44,174 [I] Checking connection..
2026-09-28 02:53:44,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:53:44,444 [I] Checking connection passed.
2026-09-28 02:53:44,444 [I] --------- Verifying version history table ---------
2026-09-28 02:53:44,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:53:44,708 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:53:44,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:53:44,949 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:53:44,950 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:53:44,950 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:53:45.201 UTC [7057] WARNING: there is no transaction in progress
2026-09-28 02:53:45,207 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:53:45,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:53:45,484 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:53:45,485 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:53:45,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:53:45,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:53:46,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s1kg.c.gcp-upstream.internal','N') ;"
2026-09-28 02:53:46,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:53:46,539 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:53:46,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:53:46,807 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:53:46,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:53:47,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:53:47,297 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:53:47,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:53:48.392 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.413 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.432 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.453 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.473 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.495 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.559 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.567 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.583 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.595 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.606 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.622 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.635 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.643 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.651 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:48.660 UTC [7165] WARNING: there is no transaction in progress
2026-09-28 02:53:49,466 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:53:49,466 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:53:49,709 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:53:49,709 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:53:49,959 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:53:49,959 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:53:50,205 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:53:50,205 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:53:50,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:53:50,731 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s1kg.c.gcp-upstream.internal';"
2026-09-28 02:53:50,973 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:53:50,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:53:51,237 [I] DB_PATCHES have already been applied
I20260928 02:53:51.243825 28567 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhPicry/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:53:51.243927 28567 mini_ranger.cc:193] Using host: 127.27.229.212
I20260928 02:53:51.259661 28567 mini_ranger.cc:227] Ranger admin URL: http://127.27.229.212:40061
Sep 28, 2026 2:53:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:53:51 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:53:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:53:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:53:51 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:53:52 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:53:52 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:53:52.454159 28567 mini_ranger.cc:160] Time spent starting Ranger: real 9.166s user 0.009s sys 0.035s
Sep 28, 2026 2:54:13 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:54:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 2:54:13 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:54:13 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:54:14 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:54:15.334275 28567 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangerkms_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/rangerkms_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.229.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.229.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:54:15 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564055, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:54:15.524008 28567 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:54:15.548691 28567 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:54:16.549314 28567 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:54:16.549508 28567 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms
I20260928 02:54:16.549678 28567 mini_ranger_kms.cc:227] Using postgres at 127.27.229.212:53067
2026-09-28 02:54:16,984 [I] DB FLAVOR :POSTGRES
2026-09-28 02:54:16,985 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:54:16,985 [I] Checking connection
2026-09-28 02:54:16,985 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:54:17,233 [I] connection success
2026-09-28 02:54:17,233 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:54:17,233 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:54:17,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:54:17,490 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:54:17,490 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:54:17,490 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:54:17,490 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:53067/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:54:17,772 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:54:18.286732 28567 mini_ranger_kms.cc:310] Created kms service
I20260928 02:54:18.481783 28567 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:54:18.524912 28567 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:54:18 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564055, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:54:18.983386 28567 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:54:18.983714 28567 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:54:18.983770 28567 mini_ranger_kms.cc:240] Using host: 127.27.229.212
I20260928 02:54:19.005334 28567 mini_ranger_kms.cc:276] Ranger KMS PID: 7346
I20260928 02:54:19.005554 28567 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.229.212:42253
Sep 28, 2026 2:54:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:54:19 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:54:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:54:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:54:20 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:54:20 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:54:21 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564061, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:21.299026 28567 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.775s user 0.122s sys 0.343s
I20260928 02:54:21.299256 28567 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:54:21.299444 28567 mini_ranger_kms.cc:401] 127.27.229.212:42253/kms/v1/keys
Sep 28 02:54:21 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564061, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
Sep 28 02:54:21 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564061, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
Sep 28 02:54:24 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:54:24.375685 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41149
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:41149
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:42253/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:54:24.728931 7396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:24.729259 7396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:24.729409 7396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:24.737008 7396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:54:24.737138 7396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:24.737218 7396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:24.737305 7396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:54:24.737406 7396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:54:24.748574 7396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.229.212:42253/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:41149
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41149
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:54:24.750000 7396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:24.751504 7396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:24.765302 7401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:24.765328 7402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:24.765648 7396 server_base.cc:1034] running on GCE node
W20260928 02:54:24.765275 7404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:24.767019 7396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:24.769495 7396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:24.770987 7396 hybrid_clock.cc:648] HybridClock initialized: now 1790564064770814 us; error 132 us; skew 500 ppm
I20260928 02:54:24.771438 7396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:54:24 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:24.779282 7396 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:54:24.781031 7396 webserver.cc:460] Webserver started at http://127.27.229.254:35313/ using document root <none> and password file <none>
I20260928 02:54:24.781704 7396 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:24.781865 7396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:24.782192 7396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:54:24 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:54:24.846540 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "f4251c20425d4ace9c0d48216b6d32d6"
format_stamp: "Formatted at 2026-09-28 02:54:24 on dist-test-slave-s1kg"
server_key: "a73d3c191240e6907fc58c0ed8b5316a"
server_key_iv: "aef4f19b67aa39619f184216513ac55a"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:24.847569 7396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "f4251c20425d4ace9c0d48216b6d32d6"
format_stamp: "Formatted at 2026-09-28 02:54:24 on dist-test-slave-s1kg"
server_key: "a73d3c191240e6907fc58c0ed8b5316a"
server_key_iv: "aef4f19b67aa39619f184216513ac55a"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:24.853369 7396 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:54:24.874534 7411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:24.876356 7396 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260928 02:54:24.876580 7396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "f4251c20425d4ace9c0d48216b6d32d6"
format_stamp: "Formatted at 2026-09-28 02:54:24 on dist-test-slave-s1kg"
server_key: "a73d3c191240e6907fc58c0ed8b5316a"
server_key_iv: "aef4f19b67aa39619f184216513ac55a"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:24.876833 7396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:24.911599 7396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:24.916107 7396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:24.916385 7396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:24.936416 7396 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:41149
I20260928 02:54:24.936614 7462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:41149 every 8 connection(s)
I20260928 02:54:24.937889 7396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:54:24.942092 7463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:24.945585 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 7396
I20260928 02:54:24.945842 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:54:24.946599 28567 external_mini_cluster.cc:1428] Setting key 8d171633386accba55efa624f29f1b40
I20260928 02:54:24.955926 7463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: Bootstrap starting.
I20260928 02:54:24.959944 7463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:24.961059 7463 log.cc:824] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:54:24 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:54:24.964578 7463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: No bootstrap required, opened a new log
I20260928 02:54:24.971524 7466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:24.948976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57266 (local address 127.27.229.254:41149)
0928 02:54:24.949851 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:24.949863 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:24.949910 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:54:24.951725 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:24.951750 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:24.952168 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:24.952644 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:24.954013 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:24.955997 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:24.957407 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:24.958170 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:24.958411 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:24.962924 (+ 4513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:24.962934 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:24.962993 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:24.963030 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:24.966748 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:24.967949 (+ 1201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:24.967957 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:24.967971 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:24.968168 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:24.968596 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:24.968599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:24.968601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:24.969176 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:24.969443 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:24.969912 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 02:54:24.970313 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":419,"threads_started":1}
I20260928 02:54:24.973654 7463 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } }
I20260928 02:54:24.974030 7463 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:24.974206 7463 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4251c20425d4ace9c0d48216b6d32d6, State: Initialized, Role: FOLLOWER
I20260928 02:54:24.974818 7463 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } }
I20260928 02:54:24.975121 7463 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:54:24.975261 7463 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:54:24.975436 7463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:24.977604 7463 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } }
I20260928 02:54:24.978154 7463 leader_election.cc:302] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4251c20425d4ace9c0d48216b6d32d6; no voters:
I20260928 02:54:24.978785 7463 leader_election.cc:288] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:54:24.979012 7468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:24.980345 7468 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [term 1 LEADER]: Becoming Leader. State: Replica: f4251c20425d4ace9c0d48216b6d32d6, State: Running, Role: LEADER
I20260928 02:54:24.981230 7463 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:54:24.981235 7468 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } }
I20260928 02:54:24.984454 7469 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4251c20425d4ace9c0d48216b6d32d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } } }
I20260928 02:54:24.984925 7469 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:24.985507 7470 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4251c20425d4ace9c0d48216b6d32d6. Latest consensus state: current_term: 1 leader_uuid: "f4251c20425d4ace9c0d48216b6d32d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4251c20425d4ace9c0d48216b6d32d6" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41149 } } }
I20260928 02:54:24.985819 7470 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6 [sys.catalog]: This master's current role is: LEADER
I20260928 02:54:24.988181 7477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:54:24.992388 7477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:54:24.993377 7463 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:54:25.009320 7477 catalog_manager.cc:1347] Generated new cluster ID: 7dce48fd49ae4a268bddd7ec883ac067
I20260928 02:54:25.009503 7477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:54:25.021889 7477 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:54:25.023379 7477 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:54:25.037228 7477 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: Generated new TSK 0
I20260928 02:54:25.038178 7477 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:54:25.851039 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:42253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:54:26 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:54:26.272784 7508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:26.273129 7508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:26.273296 7508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:26.281160 7508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:26.281319 7508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:26.281504 7508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:54:26.294004 7508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.229.212:42253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:54:26.295657 7508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:26.297065 7508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:26.323596 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:26.331833 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:26.332736 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:26.334277 7508 server_base.cc:1034] running on GCE node
I20260928 02:54:26.335206 7508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:26.336561 7508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:26.337808 7508 hybrid_clock.cc:648] HybridClock initialized: now 1790564066337754 us; error 52 us; skew 500 ppm
I20260928 02:54:26.338255 7508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:54:26 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:26.342854 7508 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:54:26.348605 7508 webserver.cc:460] Webserver started at http://127.27.229.193:37329/ using document root <none> and password file <none>
I20260928 02:54:26.349668 7508 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:26.349848 7508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:26.350253 7508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:54:26 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:54:26.372722 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "e8addcbf63864b4094bc0ec43de51bf0"
format_stamp: "Formatted at 2026-09-28 02:54:26 on dist-test-slave-s1kg"
server_key: "4cb6befd10057eaea92f020abef288ab"
server_key_iv: "ccd42884880359d28236af4483090707"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:26.373736 7508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "e8addcbf63864b4094bc0ec43de51bf0"
format_stamp: "Formatted at 2026-09-28 02:54:26 on dist-test-slave-s1kg"
server_key: "4cb6befd10057eaea92f020abef288ab"
server_key_iv: "ccd42884880359d28236af4483090707"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:26.380908 7508 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260928 02:54:26.405952 7528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:26.408013 7508 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:54:26.408236 7508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "e8addcbf63864b4094bc0ec43de51bf0"
format_stamp: "Formatted at 2026-09-28 02:54:26 on dist-test-slave-s1kg"
server_key: "4cb6befd10057eaea92f020abef288ab"
server_key_iv: "ccd42884880359d28236af4483090707"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:26.408558 7508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:26.474575 7508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:26.481628 7508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:26.482043 7508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:26.483428 7508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:26.485535 7508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:26.485693 7508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:54:26.485839 7508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:26.485915 7508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:26.598351 7508 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:46371
I20260928 02:54:26.598387 7640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:46371 every 8 connection(s)
I20260928 02:54:26.600178 7508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:54:26.602850 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 7508
I20260928 02:54:26.603084 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:54:26.603895 28567 external_mini_cluster.cc:1428] Setting key 669c94d73a2f54848305282094d8a281
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Sep 28 02:54:26 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
Sep 28 02:54:26 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:54:26.687832 7643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:26.615585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:38671 (local address 127.27.229.254:41149)
0928 02:54:26.628266 (+ 12681us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:26.628274 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:26.629135 (+ 861us) server_negotiation.cc:407] Connection header received
0928 02:54:26.646595 (+ 17460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:26.646603 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:26.646727 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:26.646902 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:26.658695 (+ 11793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:26.660059 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:26.662865 (+ 2806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:26.663513 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:26.663636 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:26.675124 (+ 11488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:26.675132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:26.675137 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:26.675185 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:26.679261 (+ 4076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:26.684189 (+ 4928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:26.684198 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:26.684203 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:26.684475 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:26.685258 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:26.685264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:26.685268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:26.685704 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:26.685849 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:26.687092 (+ 1243us) server_negotiation.cc:299] Negotiation successful
0928 02:54:26.687382 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12516,"thread_start_us":30683,"threads_started":1}
I20260928 02:54:26.691103 7644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:26.616388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41149 (local address 127.27.229.193:38671)
0928 02:54:26.628942 (+ 12554us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:26.629010 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:26.643786 (+ 14776us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:26.647675 (+ 3889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:26.647691 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:26.648559 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:26.658334 (+ 9775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:26.658375 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:26.660355 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:26.660361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:26.662637 (+ 2276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:26.662666 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:26.664014 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:26.664021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:26.664321 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:26.667095 (+ 2774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:26.667129 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:26.669923 (+ 2794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:26.683071 (+ 13148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:26.683084 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:26.683109 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:26.683899 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:26.684699 (+ 800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:26.684704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:26.684708 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:26.685093 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:26.685980 (+ 887us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:26.685990 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:26.686577 (+ 587us) client_negotiation.cc:769] Sending connection context
0928 02:54:26.688692 (+ 2115us) client_negotiation.cc:240] Negotiation successful
0928 02:54:26.689351 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12161,"thread_start_us":14295,"threads_started":1}
I20260928 02:54:26.693594 7641 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41149
I20260928 02:54:26.694000 7641 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:26.695029 7641 heartbeater.cc:511] Master 127.27.229.254:41149 requested a full tablet report, sending...
I20260928 02:54:26.698012 7428 ts_manager.cc:194] Registered new tserver with Master: e8addcbf63864b4094bc0ec43de51bf0 (127.27.229.193:46371)
I20260928 02:54:26.700121 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:38671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
W20260928 02:54:26.719048 28567 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.27.229.194: real 0.111s user 0.000s sys 0.018s
I20260928 02:54:26.720309 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:42253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564066, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
W20260928 02:54:27.259971 7647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:27.260529 7647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:27.260723 7647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:27.275964 7647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:27.276206 7647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:27.276455 7647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:54:27.295653 7647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.229.212:42253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:54:27.297922 7647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:27.299958 7647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:27.318466 7654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:27.320642 7655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:27.320871 7657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:27.322721 7647 server_base.cc:1034] running on GCE node
I20260928 02:54:27.323719 7647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:27.324932 7647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:27.326234 7647 hybrid_clock.cc:648] HybridClock initialized: now 1790564067326144 us; error 76 us; skew 500 ppm
I20260928 02:54:27.326663 7647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:27.331346 7647 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:54:27.336238 7647 webserver.cc:460] Webserver started at http://127.27.229.194:40733/ using document root <none> and password file <none>
I20260928 02:54:27.337047 7647 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:27.337201 7647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:27.337530 7647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:54:27.356289 7647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "578a841a273f4148aa3f75ca1791bac5"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "7df994cfb4976709b97572740f4dea12"
server_key_iv: "bc08456e38629f1b9fa43a0c500f6f66"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.357553 7647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "578a841a273f4148aa3f75ca1791bac5"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "7df994cfb4976709b97572740f4dea12"
server_key_iv: "bc08456e38629f1b9fa43a0c500f6f66"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.367189 7647 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20260928 02:54:27.398080 7671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:27.400234 7647 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.011s sys 0.000s
I20260928 02:54:27.400533 7647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "578a841a273f4148aa3f75ca1791bac5"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "7df994cfb4976709b97572740f4dea12"
server_key_iv: "bc08456e38629f1b9fa43a0c500f6f66"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.400892 7647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:27.427558 7647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:27.431892 7647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:27.432222 7647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:27.433238 7647 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:27.434748 7647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:27.434821 7647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:27.435024 7647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:27.435075 7647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:27.471304 7647 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:37617
I20260928 02:54:27.471321 7783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:37617 every 8 connection(s)
I20260928 02:54:27.472884 7647 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:54:27.475659 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 7647
I20260928 02:54:27.475831 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:54:27.476332 28567 external_mini_cluster.cc:1428] Setting key 57d3bee59ebd4d23935f585e2567c038
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:54:27.501675 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:27.476813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:52753 (local address 127.27.229.254:41149)
0928 02:54:27.483260 (+ 6447us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:27.483264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:27.483281 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:54:27.483332 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:27.483336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:27.483397 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:27.483533 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:27.485308 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:27.486158 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:27.488115 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:27.488715 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:27.488849 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:27.493453 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:27.493463 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:27.493472 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:27.493520 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:27.497425 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:27.498555 (+ 1130us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:27.498564 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:27.498571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:27.498732 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:27.499395 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:27.499403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:27.499410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:27.499836 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:27.500008 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:27.500924 (+ 916us) server_negotiation.cc:299] Negotiation successful
0928 02:54:27.501194 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6342,"thread_start_us":6210,"threads_started":1}
I20260928 02:54:27.502871 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:27.476273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41149 (local address 127.27.229.194:52753)
0928 02:54:27.477126 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:27.477170 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:27.480259 (+ 3089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:27.483834 (+ 3575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:27.483843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:27.484414 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:27.485122 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:27.485141 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:27.486321 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:27.486326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:27.487913 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:27.487930 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:27.488953 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:27.488963 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:27.489306 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:27.490141 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:27.490175 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:27.493151 (+ 2976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:27.497688 (+ 4537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:27.497698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:27.497718 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:27.498374 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:27.498954 (+ 580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:27.498959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:27.498962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:27.499245 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:27.499991 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:27.500001 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:27.500470 (+ 469us) client_negotiation.cc:769] Sending connection context
0928 02:54:27.500779 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:54:27.501338 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":583,"threads_started":1}
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:54:27.504554 7784 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41149
I20260928 02:54:27.504894 7784 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:27.505828 7784 heartbeater.cc:511] Master 127.27.229.254:41149 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:54:27.507737 7428 ts_manager.cc:194] Registered new tserver with Master: 578a841a273f4148aa3f75ca1791bac5 (127.27.229.194:37617)
I20260928 02:54:27.508852 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:52753
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:54:27.534780 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:42253/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:39489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:54:27.704545 7641 heartbeater.cc:503] Master 127.27.229.254:41149 was elected leader, sending a full tablet report...
W20260928 02:54:27.873721 7790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:54:27.874017 7790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:54:27.874111 7790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:54:27.881670 7790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:54:27.881789 7790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:54:27.881909 7790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:54:27.892148 7790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:39489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.229.212:42253/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41149
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:54:27.893429 7790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:54:27.894659 7790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:54:27.908674 7796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:27.908690 7798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:54:27.908927 7795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:54:27.909776 7790 server_base.cc:1034] running on GCE node
I20260928 02:54:27.910486 7790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:54:27.911639 7790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:54:27.912885 7790 hybrid_clock.cc:648] HybridClock initialized: now 1790564067912828 us; error 57 us; skew 500 ppm
I20260928 02:54:27.913343 7790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:54:27.916924 7790 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:54:27.918466 7790 webserver.cc:460] Webserver started at http://127.27.229.195:33681/ using document root <none> and password file <none>
I20260928 02:54:27.919235 7790 fs_manager.cc:359] Metadata directory not provided
I20260928 02:54:27.919377 7790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:54:27.919705 7790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:54:27 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:54:27.937507 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "607303e46ca549bb9b884899e4344f64"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "c70a2c6c1b153efd182682ce6d6819ca"
server_key_iv: "f862db6fdb32381a697fec7e77e0a4b9"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.938412 7790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "607303e46ca549bb9b884899e4344f64"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "c70a2c6c1b153efd182682ce6d6819ca"
server_key_iv: "f862db6fdb32381a697fec7e77e0a4b9"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.944492 7790 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260928 02:54:27.961481 7805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:27.963382 7790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:54:27.963582 7790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "607303e46ca549bb9b884899e4344f64"
format_stamp: "Formatted at 2026-09-28 02:54:27 on dist-test-slave-s1kg"
server_key: "c70a2c6c1b153efd182682ce6d6819ca"
server_key_iv: "f862db6fdb32381a697fec7e77e0a4b9"
server_key_version: "kuduclusterkey@0"
I20260928 02:54:27.963840 7790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:54:27.981559 7790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:54:27.987360 7790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:54:27.987742 7790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:54:27.988981 7790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:54:27.990820 7790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:54:27.990960 7790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260928 02:54:27.991099 7790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:54:27.991163 7790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:54:28.028671 7790 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:33325
I20260928 02:54:28.028708 7917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:33325 every 8 connection(s)
I20260928 02:54:28.029940 7790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:54:28.032444 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 7790
I20260928 02:54:28.032671 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:54:28.033277 28567 external_mini_cluster.cc:1428] Setting key ed200646313f14d7320ca8e4474233e0
Sep 28 02:54:28 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564067, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:54:28.052366 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.032655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:48959 (local address 127.27.229.254:41149)
0928 02:54:28.033181 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.033187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.034413 (+ 1226us) server_negotiation.cc:407] Connection header received
0928 02:54:28.036595 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.036600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.036685 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.036812 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:28.038293 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.039176 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.040753 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.041202 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.041338 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.044916 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:28.044922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:28.044925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:28.044960 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:28.048716 (+ 3756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.049760 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.049769 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.049772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.049921 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.050416 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.050421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.050425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.050711 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:28.050925 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.051694 (+ 769us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.051942 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":213,"threads_started":1}
I20260928 02:54:28.053373 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.033184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41149 (local address 127.27.229.195:48959)
0928 02:54:28.034011 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.034048 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.036317 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.037067 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.037076 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.037509 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:54:28.038116 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.038133 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.039327 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.039333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.040588 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.040606 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.041436 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.041444 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.041689 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.042417 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:54:28.042441 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:54:28.044668 (+ 2227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:54:28.048903 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:28.048914 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:28.048936 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:28.049624 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:28.050042 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:54:28.050048 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:54:28.050052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:54:28.050280 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:54:28.050887 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:54:28.050896 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:54:28.051293 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.051587 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.052177 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":355,"threads_started":1}
I20260928 02:54:28.055022 7918 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41149
I20260928 02:54:28.055545 7918 heartbeater.cc:464] Registering TS with master...
I20260928 02:54:28.056504 7918 heartbeater.cc:511] Master 127.27.229.254:41149 requested a full tablet report, sending...
I20260928 02:54:28.058166 7428 ts_manager.cc:194] Registered new tserver with Master: 607303e46ca549bb9b884899e4344f64 (127.27.229.195:33325)
I20260928 02:54:28.059120 7428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:48959
I20260928 02:54:28.063062 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:54:28.081354 7920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.067748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57282 (local address 127.27.229.254:41149)
0928 02:54:28.067979 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.067987 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.068413 (+ 426us) server_negotiation.cc:407] Connection header received
0928 02:54:28.068685 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.068691 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.068781 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.068893 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:28.070300 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.071111 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.072430 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.072868 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.073009 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.075563 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:28.075568 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:28.075572 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:28.075609 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:28.078447 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.079269 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.079276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.079279 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.079408 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.079792 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.079800 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.079803 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.080144 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:28.080390 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.080752 (+ 362us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.081015 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:54:28.086108 7428 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:54:28.093447 7428 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:54:28.134438 7932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.107160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:33325 (local address 127.0.0.1:55142)
0928 02:54:28.125671 (+ 18511us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.125690 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.125880 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.127517 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.127521 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.127549 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:28.128013 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.128024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.129894 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.129904 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.131619 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.131633 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.133336 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.133342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.133525 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.133584 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.133741 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.133934 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18362,"mutex_wait_us":165,"thread_start_us":17092,"threads_started":1}
I20260928 02:54:28.134611 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.107165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.27.229.194:37617)
0928 02:54:28.122631 (+ 15466us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.122646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.125109 (+ 2463us) server_negotiation.cc:407] Connection header received
0928 02:54:28.126119 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.126135 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.126351 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.126549 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:28.127227 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.128781 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.131889 (+ 3108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.133311 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.133479 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.133637 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.134067 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.134246 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15194,"thread_start_us":15502,"threads_started":1}
I20260928 02:54:28.135061 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.107391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.27.229.195:33325)
0928 02:54:28.125788 (+ 18397us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.125796 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.125827 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:54:28.126962 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.126971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.127213 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.127384 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:28.128219 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.129756 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.131781 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.133152 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.133336 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.133527 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.134488 (+ 961us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.134694 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16534,"thread_start_us":16340,"threads_started":1}
I20260928 02:54:28.135303 7933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.107160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37617 (local address 127.0.0.1:60850)
0928 02:54:28.124850 (+ 17690us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.124919 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.125901 (+ 982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.126554 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.126563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.126599 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:28.127036 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.127049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.128947 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.128952 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.131686 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.131700 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.133696 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.133704 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.133864 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.133892 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.133968 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.134100 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17328,"mutex_wait_us":3,"thread_start_us":18202,"threads_started":1}
I20260928 02:54:28.136252 7935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.110066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:46371 (local address 127.0.0.1:36404)
0928 02:54:28.125815 (+ 15749us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.125835 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.125963 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.127242 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.127248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.127271 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:28.127802 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.127819 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.129771 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.129778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.132274 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.132287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.135442 (+ 3155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.135450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.135610 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.135633 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.135738 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.135872 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15608,"mutex_wait_us":129,"thread_start_us":15442,"threads_started":1}
I20260928 02:54:28.136291 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.110275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36404 (local address 127.27.229.193:46371)
0928 02:54:28.126709 (+ 16434us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.126716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.126737 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:54:28.126865 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.126874 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.127045 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.127235 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:28.127969 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.129612 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.132866 (+ 3254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.134223 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.134424 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.135242 (+ 818us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.135789 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.135968 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16279,"thread_start_us":16125,"threads_started":1}
I20260928 02:54:28.139082 7719 tablet_service.cc:1467] Processing CreateTablet for tablet d696fddac31649579fc51856db96acc7 (DEFAULT_TABLE table=test-table [id=51fdc00548324631bc4f60c3acadb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:28.140884 7719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d696fddac31649579fc51856db96acc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:28.140933 7853 tablet_service.cc:1467] Processing CreateTablet for tablet d696fddac31649579fc51856db96acc7 (DEFAULT_TABLE table=test-table [id=51fdc00548324631bc4f60c3acadb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:28.142683 7853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d696fddac31649579fc51856db96acc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:28.142773 7576 tablet_service.cc:1467] Processing CreateTablet for tablet d696fddac31649579fc51856db96acc7 (DEFAULT_TABLE table=test-table [id=51fdc00548324631bc4f60c3acadb8a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:54:28.145103 7576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d696fddac31649579fc51856db96acc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:54:28.154867 7937 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Bootstrap starting.
I20260928 02:54:28.159770 7938 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Bootstrap starting.
I20260928 02:54:28.159837 7937 tablet_bootstrap.cc:654] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:28.161391 7937 log.cc:824] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:28.164052 7938 tablet_bootstrap.cc:654] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:28.165237 7937 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: No bootstrap required, opened a new log
I20260928 02:54:28.165513 7938 log.cc:824] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:28.165588 7937 ts_tablet_manager.cc:1397] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260928 02:54:28.166502 7939 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Bootstrap starting.
I20260928 02:54:28.168900 7938 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: No bootstrap required, opened a new log
I20260928 02:54:28.169255 7938 ts_tablet_manager.cc:1397] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:54:28.170732 7939 tablet_bootstrap.cc:654] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Neither blocks nor log segments found. Creating new log.
I20260928 02:54:28.172223 7939 log.cc:824] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:54:28.175886 7939 tablet_bootstrap.cc:492] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: No bootstrap required, opened a new log
I20260928 02:54:28.176239 7939 ts_tablet_manager.cc:1397] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260928 02:54:28.178411 7937 raft_consensus.cc:348] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.178997 7937 raft_consensus.cc:374] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:28.178778 7938 raft_consensus.cc:348] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.179154 7937 raft_consensus.cc:729] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 578a841a273f4148aa3f75ca1791bac5, State: Initialized, Role: FOLLOWER
I20260928 02:54:28.179319 7938 raft_consensus.cc:374] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:28.179463 7938 raft_consensus.cc:729] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607303e46ca549bb9b884899e4344f64, State: Initialized, Role: FOLLOWER
I20260928 02:54:28.179999 7937 consensus_queue.cc:260] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.180123 7938 consensus_queue.cc:260] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.181805 7784 heartbeater.cc:503] Master 127.27.229.254:41149 was elected leader, sending a full tablet report...
I20260928 02:54:28.182267 7937 ts_tablet_manager.cc:1428] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
I20260928 02:54:28.183748 7918 heartbeater.cc:503] Master 127.27.229.254:41149 was elected leader, sending a full tablet report...
I20260928 02:54:28.184433 7938 ts_tablet_manager.cc:1428] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260928 02:54:28.188052 7939 raft_consensus.cc:348] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.188570 7939 raft_consensus.cc:374] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:54:28.188735 7939 raft_consensus.cc:729] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8addcbf63864b4094bc0ec43de51bf0, State: Initialized, Role: FOLLOWER
I20260928 02:54:28.189491 7939 consensus_queue.cc:260] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.191737 7939 ts_tablet_manager.cc:1428] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260928 02:54:28.225039 7785 tablet.cc:2369] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:54:28.281677 7919 tablet.cc:2369] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:54:28.284806 7944 raft_consensus.cc:482] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:54:28.285161 7944 raft_consensus.cc:504] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.286799 7944 leader_election.cc:288] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 578a841a273f4148aa3f75ca1791bac5 (127.27.229.194:37617), e8addcbf63864b4094bc0ec43de51bf0 (127.27.229.193:46371)
I20260928 02:54:28.293498 7921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.287175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:46371 (local address 127.27.229.195:46203)
0928 02:54:28.287725 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.287757 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.287931 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.288320 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.288324 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.288352 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:28.288578 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.288590 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.289759 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.289764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.291338 (+ 1574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.291350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.292688 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.292695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.292893 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.292930 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.293052 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.293195 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387}
I20260928 02:54:28.293556 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.287410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:46203 (local address 127.27.229.193:46371)
0928 02:54:28.287603 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.287610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.287855 (+ 245us) server_negotiation.cc:407] Connection header received
0928 02:54:28.288095 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.288101 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.288191 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.288289 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:28.288723 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.289605 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.291512 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.292518 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.292648 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.292702 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.293120 (+ 418us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.293269 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:54:28.294261 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.287396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:38927 (local address 127.27.229.194:37617)
0928 02:54:28.287603 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.287609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.288234 (+ 625us) server_negotiation.cc:407] Connection header received
0928 02:54:28.288370 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.288374 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.288463 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.288563 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:54:28.289160 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.290046 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.292130 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.293178 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.293324 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.293397 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.293979 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.294085 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:54:28.294486 7946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.287265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37617 (local address 127.27.229.195:38927)
0928 02:54:28.288107 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0928 02:54:28.288124 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:54:28.288261 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:54:28.288582 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:54:28.288585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:54:28.288603 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:54:28.288991 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.289002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.290213 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.290217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.291897 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:54:28.291912 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.293438 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.293445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:54:28.293666 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.293705 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:54:28.293908 (+ 203us) client_negotiation.cc:240] Negotiation successful
0928 02:54:28.294067 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":489,"threads_started":1}
I20260928 02:54:28.294939 7596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d696fddac31649579fc51856db96acc7" candidate_uuid: "607303e46ca549bb9b884899e4344f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8addcbf63864b4094bc0ec43de51bf0" is_pre_election: true
I20260928 02:54:28.295584 7596 raft_consensus.cc:2454] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607303e46ca549bb9b884899e4344f64 in term 0.
I20260928 02:54:28.295548 7739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d696fddac31649579fc51856db96acc7" candidate_uuid: "607303e46ca549bb9b884899e4344f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578a841a273f4148aa3f75ca1791bac5" is_pre_election: true
I20260928 02:54:28.296054 7739 raft_consensus.cc:2454] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607303e46ca549bb9b884899e4344f64 in term 0.
I20260928 02:54:28.296433 7806 leader_election.cc:302] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607303e46ca549bb9b884899e4344f64, e8addcbf63864b4094bc0ec43de51bf0; no voters:
I20260928 02:54:28.296878 7944 raft_consensus.cc:2781] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:54:28.297036 7944 raft_consensus.cc:482] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:54:28.297135 7944 raft_consensus.cc:3037] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:28.299391 7944 raft_consensus.cc:504] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.300083 7944 leader_election.cc:288] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [CANDIDATE]: Term 1 election: Requested vote from peers 578a841a273f4148aa3f75ca1791bac5 (127.27.229.194:37617), e8addcbf63864b4094bc0ec43de51bf0 (127.27.229.193:46371)
I20260928 02:54:28.300619 7739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d696fddac31649579fc51856db96acc7" candidate_uuid: "607303e46ca549bb9b884899e4344f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "578a841a273f4148aa3f75ca1791bac5"
I20260928 02:54:28.300675 7596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d696fddac31649579fc51856db96acc7" candidate_uuid: "607303e46ca549bb9b884899e4344f64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8addcbf63864b4094bc0ec43de51bf0"
I20260928 02:54:28.301064 7596 raft_consensus.cc:3037] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:28.300822 7739 raft_consensus.cc:3037] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:54:28.303745 7596 raft_consensus.cc:2454] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607303e46ca549bb9b884899e4344f64 in term 1.
I20260928 02:54:28.303857 7739 raft_consensus.cc:2454] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607303e46ca549bb9b884899e4344f64 in term 1.
I20260928 02:54:28.304375 7806 leader_election.cc:302] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 607303e46ca549bb9b884899e4344f64, e8addcbf63864b4094bc0ec43de51bf0; no voters:
I20260928 02:54:28.304719 7944 raft_consensus.cc:2781] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:54:28.305198 7944 raft_consensus.cc:686] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 1 LEADER]: Becoming Leader. State: Replica: 607303e46ca549bb9b884899e4344f64, State: Running, Role: LEADER
I20260928 02:54:28.305719 7944 consensus_queue.cc:237] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } }
I20260928 02:54:28.310590 7427 catalog_manager.cc:5495] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 reported cstate change: term changed from 0 to 1, leader changed from <none> to 607303e46ca549bb9b884899e4344f64 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "607303e46ca549bb9b884899e4344f64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607303e46ca549bb9b884899e4344f64" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 33325 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 } health_report { overall_health: UNKNOWN } } }
I20260928 02:54:28.333654 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.327424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41472 (local address 127.27.229.195:33325)
0928 02:54:28.327662 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.327667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.327685 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:54:28.327955 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.327959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.328056 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.328197 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:54:28.328733 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.329484 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.331074 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.331613 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.331854 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.332356 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:54:28.332903 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:54:28.333089 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.333254 (+ 165us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.333331 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:54:28.342875 7739 raft_consensus.cc:1261] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 1 FOLLOWER]: Refusing update from remote peer 607303e46ca549bb9b884899e4344f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:28.342875 7596 raft_consensus.cc:1261] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 1 FOLLOWER]: Refusing update from remote peer 607303e46ca549bb9b884899e4344f64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:54:28.344147 7944 consensus_queue.cc:1035] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8addcbf63864b4094bc0ec43de51bf0" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:28.344522 7947 consensus_queue.cc:1035] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "578a841a273f4148aa3f75ca1791bac5" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:54:28.372134 7427 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:28.372480 7427 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57282:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:54:28.376470 7427 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4251c20425d4ace9c0d48216b6d32d6: Sending DeleteTablet for 3 replicas of tablet d696fddac31649579fc51856db96acc7
I20260928 02:54:28.377763 7719 tablet_service.cc:1514] Processing DeleteTablet for tablet d696fddac31649579fc51856db96acc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:28 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60850
I20260928 02:54:28.377770 7853 tablet_service.cc:1514] Processing DeleteTablet for tablet d696fddac31649579fc51856db96acc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:28 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:55142
I20260928 02:54:28.378535 7576 tablet_service.cc:1514] Processing DeleteTablet for tablet d696fddac31649579fc51856db96acc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:54:28 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:36404
I20260928 02:54:28.380005 7961 tablet_replica.cc:330] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: stopping tablet replica
I20260928 02:54:28.380234 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 7508
I20260928 02:54:28.380753 7961 raft_consensus.cc:2229] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:54:28.381587 7961 raft_consensus.cc:2258] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:28.382607 7960 tablet_replica.cc:330] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: stopping tablet replica
I20260928 02:54:28.383158 7960 raft_consensus.cc:2229] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:28.383697 7960 raft_consensus.cc:2258] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:28.384317 7961 ts_tablet_manager.cc:1905] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:28.386147 7960 ts_tablet_manager.cc:1905] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:54:28.393422 7960 ts_tablet_manager.cc:1918] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:28.393733 7960 log.cc:1196] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/d696fddac31649579fc51856db96acc7
I20260928 02:54:28.394016 7961 ts_tablet_manager.cc:1918] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:28.394379 7960 ts_tablet_manager.cc:1939] T d696fddac31649579fc51856db96acc7 P 578a841a273f4148aa3f75ca1791bac5: Deleting consensus metadata
I20260928 02:54:28.394348 7961 log.cc:1196] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/d696fddac31649579fc51856db96acc7
I20260928 02:54:28.395076 7961 ts_tablet_manager.cc:1939] T d696fddac31649579fc51856db96acc7 P 607303e46ca549bb9b884899e4344f64: Deleting consensus metadata
I20260928 02:54:28.397027 7415 catalog_manager.cc:4841] TS 578a841a273f4148aa3f75ca1791bac5 (127.27.229.194:37617): tablet d696fddac31649579fc51856db96acc7 (table test-table [id=51fdc00548324631bc4f60c3acadb8a1]) successfully deleted
I20260928 02:54:28.397226 7413 catalog_manager.cc:4841] TS 607303e46ca549bb9b884899e4344f64 (127.27.229.195:33325): tablet d696fddac31649579fc51856db96acc7 (table test-table [id=51fdc00548324631bc4f60c3acadb8a1]) successfully deleted
I20260928 02:54:28.398816 7962 tablet_replica.cc:330] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: stopping tablet replica
I20260928 02:54:28.399335 7962 raft_consensus.cc:2229] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:54:28.399902 7962 raft_consensus.cc:2258] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:54:28.402277 7962 ts_tablet_manager.cc:1905] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:54:28 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:54:28.409487 7962 ts_tablet_manager.cc:1918] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:54:28.409674 7962 log.cc:1196] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/d696fddac31649579fc51856db96acc7
I20260928 02:54:28.410274 7962 ts_tablet_manager.cc:1939] T d696fddac31649579fc51856db96acc7 P e8addcbf63864b4094bc0ec43de51bf0: Deleting consensus metadata
I20260928 02:54:28.412410 7412 catalog_manager.cc:4841] TS e8addcbf63864b4094bc0ec43de51bf0 (127.27.229.193:46371): tablet d696fddac31649579fc51856db96acc7 (table test-table [id=51fdc00548324631bc4f60c3acadb8a1]) successfully deleted
I20260928 02:54:28.417411 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.382046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37612 (local address 127.27.229.193:46371)
0928 02:54:28.382270 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.382278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.398068 (+ 15790us) server_negotiation.cc:407] Connection header received
0928 02:54:28.398314 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.398321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.398447 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.398569 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:28.400507 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.401617 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.403547 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.404289 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.404441 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.409909 (+ 5468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:28.409926 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:28.409948 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:28.410001 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:28.414169 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.415169 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.415179 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.415183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.415353 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.415801 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.415806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.415810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.416197 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:28.416489 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.416802 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.417049 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:54:28.418139 7636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:54:28.525118 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 7508
I20260928 02:54:28.549589 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 7647
Sep 28 02:54:28 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:54:28.567836 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.550467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56650 (local address 127.27.229.194:37617)
0928 02:54:28.550648 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.550653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.550678 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:54:28.550883 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.550889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.550977 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.551080 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:28.552746 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.553485 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.555644 (+ 2159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.556078 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.556202 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.561643 (+ 5441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:28.561659 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:28.561675 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:28.561720 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:28.564903 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.565689 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.565697 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.565699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.565830 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.566205 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.566210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.566213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.566597 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:28.566935 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.567238 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.567465 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:54:28.568578 7779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:54:28.660511 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 7647
I20260928 02:54:28.688242 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 7790
Sep 28 02:54:28 dist-test-slave-s1kg krb5kdc[6952](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:54:28.711241 7934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:54:28.689146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41482 (local address 127.27.229.195:33325)
0928 02:54:28.689743 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 02:54:28.689750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:54:28.689773 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:54:28.689836 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:54:28.689841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:54:28.689947 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:54:28.690077 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:54:28.691997 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.693504 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.695578 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:54:28.696387 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:54:28.696568 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:54:28.703283 (+ 6715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:54:28.703305 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:54:28.703327 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:54:28.703414 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:54:28.707668 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.708900 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.708910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.708913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.709130 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:54:28.709560 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:54:28.709565 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:54:28.709568 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:54:28.709916 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:54:28.710215 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0928 02:54:28.710553 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:54:28.710775 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469}
I20260928 02:54:28.712059 7913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:54:28.842146 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 7790
I20260928 02:54:28.861706 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 7396
I20260928 02:54:28.862977 7458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:54:28.976972 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 7396
I20260928 02:54:29.001956 28567 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:54:29.338066 28567 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:54:29.338285 28567 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:54:29.684571 28567 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:54:29.684 UTC [6982] LOG: received smart shutdown request
2026-09-28 02:54:29.688 UTC [6982] LOG: background worker "logical replication launcher" (PID 6991) exited with exit code 1
2026-09-28 02:54:29.688 UTC [6986] LOG: shutting down
2026-09-28 02:54:29.885 UTC [6982] LOG: database system is shut down
2026-09-28T02:54:29Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (52105 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:54:30 dist-test-slave-s1kg krb5kdc[7976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:54:30 dist-test-slave-s1kg krb5kdc[7976](info): set up 2 sockets
Sep 28 02:54:30 dist-test-slave-s1kg krb5kdc[7976](info): commencing operation
krb5kdc: starting...
W20260928 02:54:31.147293 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:54:31 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:54:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:54:31Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangeradmin_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangeradmin_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangerlookup_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangerlookup_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
I20260928 02:54:31.497829 28567 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskhPicry/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/postgres -l logfile start
2026-09-28 02:54:34.266 UTC [8006] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 02:54:34.266 UTC [8006] LOG: listening on IPv4 address "127.27.229.212", port 55635
2026-09-28 02:54:34.278 UTC [8006] LOG: listening on Unix socket "/tmp/.s.PGSQL.55635"
2026-09-28 02:54:34.307 UTC [8009] LOG: database system was shut down at 2026-09-28 02:54:32 UTC
2026-09-28 02:54:34.311 UTC [8006] LOG: database system is ready to accept connections
I20260928 02:54:35.282793 28567 mini_postgres.cc:96] Postgres bound to 55635
2026-09-28 02:54:35.300 UTC [8017] FATAL: database "slave" does not exist
127.27.229.212:55635 - accepting connections
I20260928 02:54:35.300841 28567 mini_ranger.cc:161] Starting Ranger...
I20260928 02:54:35.326787 28567 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 02:54:35.870872 28567 mini_ranger.cc:87] Created ranger Postgres database
I20260928 02:54:35.870999 28567 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin
2026-09-28 02:54:36,162 [I] DB FLAVOR :POSTGRES
2026-09-28 02:54:36,162 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:54:36,162 [I] Checking connection..
2026-09-28 02:54:36,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 02:54:36,397 [I] Checking connection passed.
2026-09-28 02:54:36,397 [I] --------- Verifying version history table ---------
2026-09-28 02:54:36,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:54:36,633 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:54:36,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:54:36,873 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 02:54:36,873 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 02:54:36,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 02:54:37.136 UTC [8081] WARNING: there is no transaction in progress
2026-09-28 02:54:37,143 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 02:54:37,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 02:54:37,419 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 02:54:37,419 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:54:37,419 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 02:54:37,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 02:54:37,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-s1kg.c.gcp-upstream.internal','N') ;"
2026-09-28 02:54:38,182 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:54:38,426 [I] Table x_portal_user does not exist in database ranger
2026-09-28 02:54:38,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:54:38,679 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 02:54:38,679 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:54:38,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:54:39,222 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 02:54:39,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 02:54:40.274 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.295 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.314 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.331 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.350 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.370 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.431 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.439 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.455 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.467 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.479 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.495 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.506 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.515 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.523 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:40.531 UTC [8189] WARNING: there is no transaction in progress
2026-09-28 02:54:41,318 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 02:54:41,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 02:54:41,574 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 02:54:41,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 02:54:41,857 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 02:54:41,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 02:54:42,123 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 02:54:42,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 02:54:42,380 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 02:54:42,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-s1kg.c.gcp-upstream.internal';"
2026-09-28 02:54:42,877 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 02:54:42,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 02:54:43,119 [I] DB_PATCHES have already been applied
I20260928 02:54:43.125567 28567 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskhPicry/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 02:54:43.125703 28567 mini_ranger.cc:193] Using host: 127.27.229.212
I20260928 02:54:43.141428 28567 mini_ranger.cc:227] Ranger admin URL: http://127.27.229.212:37755
Sep 28, 2026 2:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:54:43 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:54:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:54:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 02:54:44.272583 28567 mini_ranger.cc:160] Time spent starting Ranger: real 8.972s user 0.000s sys 0.041s
Sep 28, 2026 2:55:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:55:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:55:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:55:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:55:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 02:55:06.417816 28567 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.229.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.229.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangerkms_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.229.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/rangerkms_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.229.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.229.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.229.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/HTTP_127.27.229.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 02:55:06 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 02:55:06.608119 28567 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 02:55:06.631628 28567 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 02:55:07.690574 28567 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 02:55:07.690672 28567 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms
I20260928 02:55:07.690706 28567 mini_ranger_kms.cc:227] Using postgres at 127.27.229.212:55635
2026-09-28 02:55:08,051 [I] DB FLAVOR :POSTGRES
2026-09-28 02:55:08,051 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 02:55:08,051 [I] Checking connection
2026-09-28 02:55:08,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 02:55:08,324 [I] connection success
2026-09-28 02:55:08,325 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 02:55:08,325 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 02:55:08,325 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 02:55:08,574 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 02:55:08,574 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 02:55:08,574 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 02:55:08,574 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.229.212:55635/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 02:55:08,841 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 02:55:09.377698 28567 mini_ranger_kms.cc:310] Created kms service
I20260928 02:55:09.584553 28567 mini_ranger_kms.cc:326] Created kudu user
I20260928 02:55:09.627390 28567 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 02:55:09 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564106, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:10.076073 28567 mini_ranger_kms.cc:384] Added ranger policy
I20260928 02:55:10.076375 28567 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms:/tmp/dist-test-taskhPicry/build/asan/bin/postgresql.jar:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskhPicry/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 02:55:10.076422 28567 mini_ranger_kms.cc:240] Using host: 127.27.229.212
I20260928 02:55:10.098322 28567 mini_ranger_kms.cc:276] Ranger KMS PID: 8370
I20260928 02:55:10.098474 28567 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.229.212:54905
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:55:11 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 28 02:55:12 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:12.445665 28567 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.838s user 0.062s sys 0.319s
I20260928 02:55:12.445914 28567 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 02:55:12.446090 28567 mini_ranger_kms.cc:401] 127.27.229.212:54905/kms/v1/keys
Sep 28 02:55:12 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
Sep 28 02:55:13 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564112, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.229.212@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
Sep 28 02:55:15 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:55:15.652709 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34893
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34893
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:16.011124 8420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:16.011488 8420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:16.011665 8420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:16.019255 8420 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:16.019430 8420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:16.019556 8420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:16.019659 8420 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:16.019757 8420 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:16.030174 8420 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.229.212:54905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34893
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:16.031626 8420 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:16.033011 8420 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:16.045552 8425 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:16.045656 8428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:16.045666 8426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:16.046418 8420 server_base.cc:1034] running on GCE node
I20260928 02:55:16.047263 8420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:16.048776 8420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:16.050026 8420 hybrid_clock.cc:648] HybridClock initialized: now 1790564116049987 us; error 37 us; skew 500 ppm
I20260928 02:55:16.050443 8420 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:16 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:16.057276 8420 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:16.059140 8420 webserver.cc:460] Webserver started at http://127.27.229.254:43963/ using document root <none> and password file <none>
I20260928 02:55:16.059818 8420 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:16.059918 8420 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:16.060189 8420 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:55:16 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564116, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:16.122001 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "07e2e38bdcea4752be24b6228794b924"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-s1kg"
server_key: "b9b14ba6839dac0ce2cca2093c4829f8"
server_key_iv: "689d054414d3c10e92dc971c44869de1"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:16.123003 8420 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "07e2e38bdcea4752be24b6228794b924"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-s1kg"
server_key: "b9b14ba6839dac0ce2cca2093c4829f8"
server_key_iv: "689d054414d3c10e92dc971c44869de1"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:16.128955 8420 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:55:16.149803 8435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:16.151762 8420 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:55:16.151973 8420 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "07e2e38bdcea4752be24b6228794b924"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-s1kg"
server_key: "b9b14ba6839dac0ce2cca2093c4829f8"
server_key_iv: "689d054414d3c10e92dc971c44869de1"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:16.152249 8420 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:16.178751 8420 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:16.184094 8420 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:16.184440 8420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:16.205271 8486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34893 every 8 connection(s)
I20260928 02:55:16.205293 8420 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34893
I20260928 02:55:16.207240 8420 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:16.209513 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 8420
I20260928 02:55:16.209756 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:55:16.210601 28567 external_mini_cluster.cc:1428] Setting key 939b618ca9b78626c8e68823166203d2
I20260928 02:55:16.213129 8487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:55:16 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:16.225530 8487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Bootstrap starting.
I20260928 02:55:16.229059 8487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:16.230121 8487 log.cc:824] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:16.233632 8487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: No bootstrap required, opened a new log
I20260928 02:55:16.234230 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:16.212869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34414 (local address 127.27.229.254:34893)
0928 02:55:16.213779 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:16.213795 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:16.213844 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:55:16.215973 (+ 2129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:16.216004 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:16.216398 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:16.216843 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:16.218367 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.219826 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.221116 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:16.221803 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:16.222057 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:16.226365 (+ 4308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:16.226373 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:16.226387 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:16.226424 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:16.229661 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.230404 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.230412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.230419 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.230635 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:16.231160 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:16.231163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:16.231166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:16.231822 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:16.232125 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:16.232527 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 02:55:16.232941 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":412,"threads_started":1}
I20260928 02:55:16.241744 8487 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:16.242105 8487 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:16.242182 8487 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e2e38bdcea4752be24b6228794b924, State: Initialized, Role: FOLLOWER
I20260928 02:55:16.242667 8487 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:16.242965 8487 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:16.243132 8487 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:16.243335 8487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:16.245540 8487 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:16.245985 8487 leader_election.cc:302] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e2e38bdcea4752be24b6228794b924; no voters:
I20260928 02:55:16.246575 8487 leader_election.cc:288] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:16.246793 8492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:16.247599 8492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 LEADER]: Becoming Leader. State: Replica: 07e2e38bdcea4752be24b6228794b924, State: Running, Role: LEADER
I20260928 02:55:16.248265 8492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:16.248808 8487 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:16.251519 8494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e2e38bdcea4752be24b6228794b924. Latest consensus state: current_term: 1 leader_uuid: "07e2e38bdcea4752be24b6228794b924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } } }
I20260928 02:55:16.251577 8493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07e2e38bdcea4752be24b6228794b924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } } }
I20260928 02:55:16.251904 8494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:16.251987 8493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:16.259639 8499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:16.259902 8487 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:55:16.272053 8499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:16.285189 8499 catalog_manager.cc:1347] Generated new cluster ID: 74912d79f59942df8b9a722fc8bcdbf1
I20260928 02:55:16.285322 8499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:16.304636 8499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:16.306670 8499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:16.315650 8499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Generated new TSK 0
I20260928 02:55:16.316767 8499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:55:17.107615 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:17 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 02:55:17.551668 8532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:17.552081 8532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:17.552304 8532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:17.564353 8532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:17.564574 8532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:17.564780 8532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:17.581375 8532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:17.583371 8532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:17.585171 8532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:17.603192 8542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:17.603214 8543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:17.608999 8532 server_base.cc:1034] running on GCE node
W20260928 02:55:17.609334 8545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:17.610201 8532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:17.618428 8532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:17.621999 8532 hybrid_clock.cc:648] HybridClock initialized: now 1790564117621859 us; error 126 us; skew 500 ppm
I20260928 02:55:17.622565 8532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:17 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:17.627410 8532 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:17.640651 8532 webserver.cc:460] Webserver started at http://127.27.229.193:45527/ using document root <none> and password file <none>
I20260928 02:55:17.642179 8532 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:17.642462 8532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:17.642966 8532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:55:17 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:17.661821 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-s1kg"
server_key: "07c2d6353e627a3cb4745fcc329667ac"
server_key_iv: "79f6fb41b2323ef4c3e55d197ba5dd86"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:17.663004 8532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-s1kg"
server_key: "07c2d6353e627a3cb4745fcc329667ac"
server_key_iv: "79f6fb41b2323ef4c3e55d197ba5dd86"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:17.670544 8532 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260928 02:55:17.703586 8552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:17.705858 8532 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.009s sys 0.008s
I20260928 02:55:17.706162 8532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-s1kg"
server_key: "07c2d6353e627a3cb4745fcc329667ac"
server_key_iv: "79f6fb41b2323ef4c3e55d197ba5dd86"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:17.706437 8532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:17.750777 8532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:17.756554 8532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:17.756930 8532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:17.758266 8532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:17.763899 8532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:17.764058 8532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:17.764232 8532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:17.764310 8532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:55:17 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:17.946125 8532 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37939
I20260928 02:55:17.946146 8664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37939 every 8 connection(s)
I20260928 02:55:17.947994 8532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:17.950237 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 8532
I20260928 02:55:17.950511 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:55:17.951525 28567 external_mini_cluster.cc:1428] Setting key 2de8fc1f144850169e5e75e618bc4d86
Sep 28 02:55:17 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:55:18.015712 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:17.951175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:45159 (local address 127.27.229.254:34893)
0928 02:55:17.980401 (+ 29226us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:17.980408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:17.980437 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:55:17.985334 (+ 4897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:17.985345 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:17.985498 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:17.985610 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:17.998595 (+ 12985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:17.999748 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.002549 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.003218 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.003333 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:18.007557 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:18.007563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:18.007566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:18.007610 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:18.011391 (+ 3781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:18.012489 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:18.012496 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:18.012499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:18.012703 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:18.013414 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:18.013420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:18.013423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:18.013834 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:18.013994 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:18.015004 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0928 02:55:18.015292 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29072,"thread_start_us":29075,"threads_started":1}
I20260928 02:55:18.017051 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:17.952342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.193:45159)
0928 02:55:17.973947 (+ 21605us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:17.974010 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:17.983135 (+ 9125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:17.986440 (+ 3305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:17.986460 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:17.987432 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:17.988576 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:17.988609 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.000112 (+ 11503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.000118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:18.002293 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:18.002327 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.003735 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.003742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:18.004086 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:18.004942 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:18.004968 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:18.007264 (+ 2296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:18.011596 (+ 4332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:18.011607 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:18.011629 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:18.012320 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:18.012874 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:18.012918 (+ 44us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:18.012924 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:18.013259 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:18.014008 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:18.014014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:18.014547 (+ 533us) client_negotiation.cc:769] Sending connection context
0928 02:55:18.014797 (+ 250us) client_negotiation.cc:240] Negotiation successful
0928 02:55:18.015463 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21224,"thread_start_us":26451,"threads_started":1}
I20260928 02:55:18.019042 8665 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:18.019431 8665 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:18.020429 8665 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:18.023291 8452 ts_manager.cc:194] Registered new tserver with Master: a0b5d56faf4a452aba5ed26d4a6e7da4 (127.27.229.193:37939)
I20260928 02:55:18.024824 8452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:45159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:55:18.051651 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:18 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564117, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
W20260928 02:55:18.483834 8671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:18.484236 8671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:18.484418 8671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:18.494580 8671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:18.494853 8671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:18.495082 8671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:18.508620 8671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:18.510289 8671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:18.512003 8671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:18.526774 8686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:18.527631 8685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:18.527673 8688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:18.528023 8671 server_base.cc:1034] running on GCE node
I20260928 02:55:18.528827 8671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:18.530184 8671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:18.531486 8671 hybrid_clock.cc:648] HybridClock initialized: now 1790564118531423 us; error 52 us; skew 500 ppm
I20260928 02:55:18.531908 8671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:18 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:18.536116 8671 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:18.537910 8671 webserver.cc:460] Webserver started at http://127.27.229.194:46065/ using document root <none> and password file <none>
I20260928 02:55:18.538650 8671 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:18.538812 8671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:18.539194 8671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:55:18 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:18.556998 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "ada60dc825be49369b058b83772b08d6"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-s1kg"
server_key: "888b11741a0da842efd9c3fda95d8a0c"
server_key_iv: "aea0175e5edb212848d31a2b4ee8e869"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:18.558089 8671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "ada60dc825be49369b058b83772b08d6"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-s1kg"
server_key: "888b11741a0da842efd9c3fda95d8a0c"
server_key_iv: "aea0175e5edb212848d31a2b4ee8e869"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:18.564206 8671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260928 02:55:18.582450 8695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:18.584468 8671 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:55:18.584688 8671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "ada60dc825be49369b058b83772b08d6"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-s1kg"
server_key: "888b11741a0da842efd9c3fda95d8a0c"
server_key_iv: "aea0175e5edb212848d31a2b4ee8e869"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:18.584954 8671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:18.622696 8671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:18.628082 8671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:18.628419 8671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:18.629663 8671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:18.631527 8671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:18.631632 8671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:18.631745 8671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:18.631806 8671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:18.669616 8671 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:33501
I20260928 02:55:18.669674 8807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:33501 every 8 connection(s)
I20260928 02:55:18.671310 8671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:18.680154 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 8671
I20260928 02:55:18.680361 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:55:18.680812 28567 external_mini_cluster.cc:1428] Setting key a2a13b5e30278268c5f3e9d78377a026
Sep 28 02:55:18 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564118, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:18.695550 8810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:18.674441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:51041 (local address 127.27.229.254:34893)
0928 02:55:18.675018 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:18.675027 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:18.676220 (+ 1193us) server_negotiation.cc:407] Connection header received
0928 02:55:18.678040 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:18.678046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:18.678140 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:18.678267 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:18.679852 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.680976 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.683001 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.683622 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.683757 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:18.688581 (+ 4824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:18.688589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:18.688593 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:18.688639 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:18.692200 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:18.693012 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:18.693017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:18.693021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:18.693166 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:18.693647 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:18.693652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:18.693654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:18.693939 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:18.694113 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:18.694903 (+ 790us) server_negotiation.cc:299] Negotiation successful
0928 02:55:18.695168 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":247,"threads_started":1}
I20260928 02:55:18.696739 8811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:18.675153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.194:51041)
0928 02:55:18.675987 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:18.676036 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:18.677756 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:18.678478 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:18.678486 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:18.679020 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:18.679667 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:18.679684 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.681161 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.681167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:18.682774 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:18.682793 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:18.683882 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:18.683890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:18.684236 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:18.685104 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:18.685134 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:18.688205 (+ 3071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:18.692377 (+ 4172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:18.692391 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:18.692405 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:18.692868 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:18.693285 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:18.693290 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:18.693295 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:18.693502 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:18.694084 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:18.694093 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:18.694446 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 02:55:18.694755 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:55:18.695340 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":371,"threads_started":1}
I20260928 02:55:18.698226 8808 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:18.698573 8808 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:18.699384 8808 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:18.700888 8452 ts_manager.cc:194] Registered new tserver with Master: ada60dc825be49369b058b83772b08d6 (127.27.229.194:33501)
I20260928 02:55:18.701714 8452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:51041
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:55:18.742756 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:19.029428 8665 heartbeater.cc:503] Master 127.27.229.254:34893 was elected leader, sending a full tablet report...
W20260928 02:55:19.145536 8814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:19.145905 8814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:19.146080 8814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:19.153879 8814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:19.154045 8814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:19.154232 8814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:19.164503 8814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:19.165998 8814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:19.167550 8814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:19.184362 8819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:19.184445 8822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:19.184489 8820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:19.186008 8814 server_base.cc:1034] running on GCE node
I20260928 02:55:19.186637 8814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:19.187733 8814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:19.188931 8814 hybrid_clock.cc:648] HybridClock initialized: now 1790564119188882 us; error 38 us; skew 500 ppm
I20260928 02:55:19.189311 8814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:19.192731 8814 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:19.194417 8814 webserver.cc:460] Webserver started at http://127.27.229.195:34095/ using document root <none> and password file <none>
I20260928 02:55:19.195246 8814 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:19.195399 8814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:19.195720 8814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:19.214310 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "cd0be9f0af9f49f0a3f10c44a91788d0"
format_stamp: "Formatted at 2026-09-28 02:55:19 on dist-test-slave-s1kg"
server_key: "e896bb2721e1a45673d4f9302c12dcd6"
server_key_iv: "8cb75b7d102f99637079e61c19f88c97"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:19.215462 8814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "cd0be9f0af9f49f0a3f10c44a91788d0"
format_stamp: "Formatted at 2026-09-28 02:55:19 on dist-test-slave-s1kg"
server_key: "e896bb2721e1a45673d4f9302c12dcd6"
server_key_iv: "8cb75b7d102f99637079e61c19f88c97"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:19.221769 8814 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:55:19.238651 8829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:19.240484 8814 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:55:19.240727 8814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "cd0be9f0af9f49f0a3f10c44a91788d0"
format_stamp: "Formatted at 2026-09-28 02:55:19 on dist-test-slave-s1kg"
server_key: "e896bb2721e1a45673d4f9302c12dcd6"
server_key_iv: "8cb75b7d102f99637079e61c19f88c97"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:19.240993 8814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:19.267321 8814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:19.272208 8814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:19.272557 8814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:19.273602 8814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:19.275153 8814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:19.275267 8814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:19.275422 8814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:19.275506 8814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:19.312870 8814 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43631
I20260928 02:55:19.312906 8941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43631 every 8 connection(s)
I20260928 02:55:19.314304 8814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:19.315716 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 8814
I20260928 02:55:19.315908 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:55:19.316483 28567 external_mini_cluster.cc:1428] Setting key c2bc910d0bcb8e7c59fed31a0638f6fc
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564119, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:19.336575 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:19.317712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:41963 (local address 127.27.229.254:34893)
0928 02:55:19.318326 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:19.318335 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:19.319083 (+ 748us) server_negotiation.cc:407] Connection header received
0928 02:55:19.321173 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:19.321177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:19.321258 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:19.321357 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:19.323152 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.323966 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.325742 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.326213 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.326328 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:19.330026 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:19.330033 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:19.330037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:19.330074 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:19.333463 (+ 3389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.334243 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.334250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.334254 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.334383 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.334861 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.334868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.334874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.335160 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:19.335315 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:19.335987 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 02:55:19.336209 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":325,"threads_started":1}
I20260928 02:55:19.337586 8945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:19.317986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.195:41963)
0928 02:55:19.318923 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:19.318961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:19.320895 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:19.321632 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:19.321640 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:19.322099 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:19.322713 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:19.322731 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.324154 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.324157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:19.325596 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:19.325608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.326433 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.326438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:19.326739 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:19.327505 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:19.327528 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:19.329761 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:19.333640 (+ 3879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:19.333650 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:19.333676 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:19.334103 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:19.334498 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:19.334503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:19.334507 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:19.334726 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:19.335287 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:19.335295 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:19.335645 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 02:55:19.335914 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 02:55:19.336456 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":380,"threads_started":1}
I20260928 02:55:19.339208 8942 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:19.339515 8942 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:19.340235 8942 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:19.341710 8452 ts_manager.cc:194] Registered new tserver with Master: cd0be9f0af9f49f0a3f10c44a91788d0 (127.27.229.195:43631)
I20260928 02:55:19.342550 8452 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:41963
I20260928 02:55:19.345733 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:55:19.347750 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 8532
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:55:19.370436 8947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:19.348811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48130 (local address 127.27.229.193:37939)
0928 02:55:19.349529 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:19.349536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:19.349564 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:55:19.349624 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:19.349630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:19.349932 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:19.350070 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:19.351637 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.352624 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.354271 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.354995 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.355101 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:19.360598 (+ 5497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:19.360614 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:19.360630 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:19.360672 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:19.365484 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.366763 (+ 1279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.366773 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.366777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.367029 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.367598 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.367604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.367627 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.368132 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:19.368430 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:19.369726 (+ 1296us) server_negotiation.cc:299] Negotiation successful
0928 02:55:19.370008 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":388,"threads_started":1}
I20260928 02:55:19.372944 8660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:19.457456 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 8532
I20260928 02:55:19.473690 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 8671
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:55:19.491185 8949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:19.474769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36880 (local address 127.27.229.194:33501)
0928 02:55:19.475367 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:19.475373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:19.475397 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:55:19.475457 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:19.475464 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:19.475678 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:19.475855 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:19.477102 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.478073 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.479522 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.480084 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.480234 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:19.485041 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:19.485053 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:19.485069 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:19.485119 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:19.488141 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.488951 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.488959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.488961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.489103 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.489563 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.489566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.489568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.489971 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:19.490216 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:19.490528 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:55:19.490764 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20260928 02:55:19.492445 8803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:19.576412 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 8671
I20260928 02:55:19.596807 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 8814
Sep 28 02:55:19 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564115, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:55:19.613862 8951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:19.597704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58410 (local address 127.27.229.195:43631)
0928 02:55:19.598326 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:19.598334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:19.598369 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:55:19.598429 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:19.598435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:19.598653 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:19.598832 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:19.600204 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.601247 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.602570 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:19.603309 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:19.603464 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:19.608035 (+ 4571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:19.608050 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:19.608065 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:19.608105 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:19.610999 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.611791 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.611798 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.611800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.611941 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:19.612399 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:19.612403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:19.612405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:19.612757 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:19.613015 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:19.613345 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:55:19.613556 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":261,"threads_started":1}
I20260928 02:55:19.615422 8937 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:19.700904 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 8814
I20260928 02:55:19.717195 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 8420
I20260928 02:55:19.718504 8482 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:19.824153 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 8420
I20260928 02:55:19.849328 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34893
--webserver_interface=127.27.229.254
--webserver_port=43963
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:34893
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:20.182595 8954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:20.183003 8954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:20.183182 8954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:20.190714 8954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:20.190891 8954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:20.190990 8954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:20.191110 8954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:20.191215 8954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:20.201681 8954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:34893
--ranger_config_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=43963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:20.203099 8954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:20.204486 8954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:20.216905 8959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.216974 8960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.217001 8962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.217844 8954 server_base.cc:1034] running on GCE node
I20260928 02:55:20.218736 8954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:20.220273 8954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:20.221549 8954 hybrid_clock.cc:648] HybridClock initialized: now 1790564120221455 us; error 55 us; skew 500 ppm
I20260928 02:55:20.221877 8954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:20 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:20.225325 8954 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:20.227072 8954 webserver.cc:460] Webserver started at http://127.27.229.254:43963/ using document root <none> and password file <none>
I20260928 02:55:20.227897 8954 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:20.228009 8954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:55:20 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:20.293548 8954 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:55:20.296954 8970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:20.298636 8954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:55:20.298893 8954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "07e2e38bdcea4752be24b6228794b924"
format_stamp: "Formatted at 2026-09-28 02:55:16 on dist-test-slave-s1kg"
server_key: "b9b14ba6839dac0ce2cca2093c4829f8"
server_key_iv: "689d054414d3c10e92dc971c44869de1"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:20.299933 8954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:20.323398 8954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:20.327651 8954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:20.328042 8954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:20.347332 8954 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:34893
I20260928 02:55:20.347357 9021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:34893 every 8 connection(s)
I20260928 02:55:20.348869 8954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:20.354043 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 8954
I20260928 02:55:20.354996 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:37939
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:34893
--webserver_port=45527
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:20.356241 9022 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:55:20.359786 9022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Bootstrap starting.
I20260928 02:55:20.379148 9022 log.cc:824] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:20.388507 9022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:55:20.389091 9022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Bootstrap complete.
I20260928 02:55:20.398164 9022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:20.399173 9022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07e2e38bdcea4752be24b6228794b924, State: Initialized, Role: FOLLOWER
I20260928 02:55:20.399878 9022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:20.400130 9022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:20.400270 9022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:20.400468 9022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:55:20.404825 9022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:20.405459 9022 leader_election.cc:302] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07e2e38bdcea4752be24b6228794b924; no voters:
I20260928 02:55:20.406148 9022 leader_election.cc:288] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:55:20.406296 9026 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:55:20.408753 9022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:20.413424 9026 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [term 2 LEADER]: Becoming Leader. State: Replica: 07e2e38bdcea4752be24b6228794b924, State: Running, Role: LEADER
I20260928 02:55:20.414137 9026 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } }
I20260928 02:55:20.417059 9027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "07e2e38bdcea4752be24b6228794b924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } } }
I20260928 02:55:20.417544 9027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:20.418267 9028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07e2e38bdcea4752be24b6228794b924. Latest consensus state: current_term: 2 leader_uuid: "07e2e38bdcea4752be24b6228794b924" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07e2e38bdcea4752be24b6228794b924" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 34893 } } }
I20260928 02:55:20.418632 9028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:20.423619 9037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:20.429687 9037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:20.430650 9022 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 02:55:20.431160 9037 catalog_manager.cc:1259] Loaded cluster ID: 74912d79f59942df8b9a722fc8bcdbf1
I20260928 02:55:20.431264 9037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:20.435302 9037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:20.436830 9037 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Loaded TSK: 0
I20260928 02:55:20.438437 9037 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:55:20.793366 9024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:20.793807 9024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:20.793977 9024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:20.806116 9024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:20.806298 9024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:20.806491 9024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:20.820248 9024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:37939
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=45527
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:20.821933 9024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:20.823549 9024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:20.875731 9060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.878034 9061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:20.892999 9063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:20.893545 9024 server_base.cc:1034] running on GCE node
I20260928 02:55:20.894377 9024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:20.895651 9024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:20.896920 9024 hybrid_clock.cc:648] HybridClock initialized: now 1790564120896844 us; error 65 us; skew 500 ppm
I20260928 02:55:20.897284 9024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:20 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:20.901512 9024 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:20.909912 9024 webserver.cc:460] Webserver started at http://127.27.229.193:45527/ using document root <none> and password file <none>
I20260928 02:55:20.910910 9024 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:20.911090 9024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:55:20 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:20.967716 9024 fs_manager.cc:709] Time spent opening directory manager: real 0.024s user 0.004s sys 0.012s
I20260928 02:55:20.972219 9071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:20.974282 9024 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260928 02:55:20.974561 9024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4"
format_stamp: "Formatted at 2026-09-28 02:55:17 on dist-test-slave-s1kg"
server_key: "07c2d6353e627a3cb4745fcc329667ac"
server_key_iv: "79f6fb41b2323ef4c3e55d197ba5dd86"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:20.976213 9024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:21.057895 9024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:21.068096 9024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:21.068449 9024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:21.070477 9024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:21.072511 9024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:21.072634 9024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:21.072760 9024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:21.072846 9024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:21.134474 9024 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:37939
I20260928 02:55:21.136523 9024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:21.137069 9184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:37939 every 8 connection(s)
I20260928 02:55:21.143313 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9024
I20260928 02:55:21.144161 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:33501
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:34893
--webserver_port=46065
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:21 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564120, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:21.179454 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.146570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:58791 (local address 127.27.229.254:34893)
0928 02:55:21.150990 (+ 4420us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.151021 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.151076 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:55:21.151553 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.151577 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.152050 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.152473 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:21.154255 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.156359 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.158868 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.160024 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.160237 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.169059 (+ 8822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:21.169075 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:21.169102 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:21.169178 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:21.173695 (+ 4517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:21.174749 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:21.174759 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:21.174767 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:21.175038 (+ 271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:21.175650 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:21.175657 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:21.175664 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:21.176372 (+ 708us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:21.176604 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.177517 (+ 913us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.177953 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4118,"thread_start_us":3335,"threads_started":1}
I20260928 02:55:21.180439 9187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.145403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.193:58791)
0928 02:55:21.146356 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:21.146419 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:21.149505 (+ 3086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:21.152642 (+ 3137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:21.152656 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:21.153216 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:21.153809 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:21.153828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.156584 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.156591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:21.158641 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:21.158660 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.160311 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.160319 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:21.163205 (+ 2886us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.164188 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:21.164229 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:21.168637 (+ 4408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:21.173908 (+ 5271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:21.173918 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:21.173935 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:21.174552 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:21.175236 (+ 684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:21.175241 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:21.175245 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:21.175503 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:21.176534 (+ 1031us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:21.176543 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:21.176948 (+ 405us) client_negotiation.cc:769] Sending connection context
0928 02:55:21.177257 (+ 309us) client_negotiation.cc:240] Negotiation successful
0928 02:55:21.177821 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":1331,"threads_started":1}
I20260928 02:55:21.183954 9185 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:21.184310 9185 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:21.185199 9185 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
W20260928 02:55:21.188444 9185 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:34893 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:55:21.188673 9185 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:21.188961 9185 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:21.190155 9185 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:21.190423 9185 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
W20260928 02:55:21.191488 9185 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:55:21.556136 9189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:21.556586 9189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:21.556806 9189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:21.567206 9189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:21.567440 9189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:21.567636 9189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:21.578299 9189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:33501
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=46065
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:21.579816 9189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:21.581293 9189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:21.597395 9204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:21.597985 9203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:21.599128 9206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:21.601096 9189 server_base.cc:1034] running on GCE node
I20260928 02:55:21.601922 9189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:21.603283 9189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:21.604535 9189 hybrid_clock.cc:648] HybridClock initialized: now 1790564121604440 us; error 95 us; skew 500 ppm
I20260928 02:55:21.604964 9189 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:21 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:21.609408 9189 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:21.611546 9189 webserver.cc:460] Webserver started at http://127.27.229.194:46065/ using document root <none> and password file <none>
I20260928 02:55:21.612581 9189 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:21.612757 9189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:55:21 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:21.653249 9189 fs_manager.cc:709] Time spent opening directory manager: real 0.018s user 0.006s sys 0.009s
I20260928 02:55:21.663780 9214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:21.665946 9189 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260928 02:55:21.666251 9189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "ada60dc825be49369b058b83772b08d6"
format_stamp: "Formatted at 2026-09-28 02:55:18 on dist-test-slave-s1kg"
server_key: "888b11741a0da842efd9c3fda95d8a0c"
server_key_iv: "aea0175e5edb212848d31a2b4ee8e869"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:21.667908 9189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:21.720867 9189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:21.726617 9189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:21.726980 9189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:21.728216 9189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:21.730072 9189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:21.730189 9189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:21.730319 9189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:21.730404 9189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 02:55:21 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:21.782799 9189 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:33501
I20260928 02:55:21.782989 9328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:33501 every 8 connection(s)
I20260928 02:55:21.784847 9189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:21.794505 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9189
I20260928 02:55:21.795305 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.229.212:54905/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:43631
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:34893
--webserver_port=34095
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:21 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:21.842931 9331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.798066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:34645 (local address 127.27.229.254:34893)
0928 02:55:21.819297 (+ 21231us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:21.819305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:21.819334 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:55:21.819413 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:21.819419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:21.819516 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:21.819669 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:21.822429 (+ 2760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.824833 (+ 2404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.831435 (+ 6602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.832326 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.832499 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.836449 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:21.836454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:21.836457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:21.836507 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:21.839635 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:21.840458 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:21.840463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:21.840469 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:21.840596 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:21.841077 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:21.841082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:21.841087 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:21.841380 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:21.841546 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:21.842280 (+ 734us) server_negotiation.cc:299] Negotiation successful
0928 02:55:21.842510 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21062,"thread_start_us":22925,"threads_started":1}
I20260928 02:55:21.844051 9332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:21.798742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.194:34645)
0928 02:55:21.805504 (+ 6762us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:21.805585 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:21.808870 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:21.820281 (+ 11411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:21.820297 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:21.821022 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:21.822126 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:21.822155 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.828404 (+ 6249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.828412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:21.831151 (+ 2739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:21.831175 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:21.832668 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:21.832678 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:21.833021 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:21.833851 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:21.833875 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:21.836154 (+ 2279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:21.839813 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:21.839823 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:21.839843 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:21.840312 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:21.840720 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:21.840724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:21.840728 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:21.840951 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:21.841525 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:21.841534 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:21.841909 (+ 375us) client_negotiation.cc:769] Sending connection context
0928 02:55:21.842188 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:55:21.842766 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6247,"thread_start_us":11658,"threads_started":1}
I20260928 02:55:21.845762 9329 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:21.846078 9329 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:21.847019 9329 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:21.848857 8987 ts_manager.cc:194] Registered new tserver with Master: ada60dc825be49369b058b83772b08d6 (127.27.229.194:33501)
I20260928 02:55:21.850502 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:34645
I20260928 02:55:22.191645 9185 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:22.192118 9185 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:22.193519 8987 ts_manager.cc:194] Registered new tserver with Master: a0b5d56faf4a452aba5ed26d4a6e7da4 (127.27.229.193:37939)
I20260928 02:55:22.194779 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:58791
W20260928 02:55:22.272748 9333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:22.273185 9333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
Sep 28 02:55:22 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
W20260928 02:55:22.273386 9333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:22.284099 9333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:22.284257 9333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:22.284518 9333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:22.300261 9333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.229.212:54905/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:43631
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=34095
--tserver_master_addrs=127.27.229.254:34893
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:22.302325 9333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:22.304164 9333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:22.334460 9340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:22.335640 9339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:22.344091 9343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:22.346361 9333 server_base.cc:1034] running on GCE node
I20260928 02:55:22.347266 9333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:22.363054 9333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:22.364439 9333 hybrid_clock.cc:648] HybridClock initialized: now 1790564122364374 us; error 76 us; skew 500 ppm
I20260928 02:55:22.364841 9333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:22 dist-test-slave-s1kg krb5kdc[7976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:22.368316 9333 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:22.372360 9333 webserver.cc:460] Webserver started at http://127.27.229.195:34095/ using document root <none> and password file <none>
I20260928 02:55:22.373208 9333 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:22.373440 9333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 02:55:22 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:22.409415 9333 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.000s sys 0.012s
I20260928 02:55:22.413354 9350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:22.415339 9333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 02:55:22.415524 9333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "cd0be9f0af9f49f0a3f10c44a91788d0"
format_stamp: "Formatted at 2026-09-28 02:55:19 on dist-test-slave-s1kg"
server_key: "e896bb2721e1a45673d4f9302c12dcd6"
server_key_iv: "8cb75b7d102f99637079e61c19f88c97"
server_key_version: "kuduclusterkey@0"
I20260928 02:55:22.416913 9333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:22.442538 9333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:22.447844 9333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:22.448195 9333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:22.449396 9333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:22.451141 9333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:22.451248 9333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:22.451355 9333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:22.451427 9333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:22.492633 9333 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43631
I20260928 02:55:22.492715 9462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43631 every 8 connection(s)
I20260928 02:55:22.494438 9333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:22.503960 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9333
Sep 28 02:55:22 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564122, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:22.539922 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.505683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50120 (local address 127.27.229.254:34893)
0928 02:55:22.516572 (+ 10889us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.516578 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.516599 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:55:22.516671 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.516676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.516759 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.516865 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:22.520331 (+ 3466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.521544 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.524081 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.525682 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.525824 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.529765 (+ 3941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:22.529772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:22.529776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:22.529818 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:22.535639 (+ 5821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.536666 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.536672 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.536676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.536856 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.537347 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.537352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.537356 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.537747 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:22.538033 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.539258 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.539502 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10769,"thread_start_us":9397,"threads_started":1}
I20260928 02:55:22.540529 9468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.498509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:39039 (local address 127.27.229.254:34893)
0928 02:55:22.515362 (+ 16853us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.515369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.520140 (+ 4771us) server_negotiation.cc:407] Connection header received
0928 02:55:22.522224 (+ 2084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.522230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.522322 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.522421 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:22.524586 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.525716 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.528275 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.528946 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.529048 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.533512 (+ 4464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:22.533519 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:22.533523 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:22.533572 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:22.536942 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.537868 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.537872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.537875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.538000 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.538542 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.538547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.538550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.538874 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:22.538989 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.539944 (+ 955us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.540167 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16715,"thread_start_us":17807,"threads_started":1}
I20260928 02:55:22.542953 9465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.497919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:34893 (local address 127.27.229.195:39039)
0928 02:55:22.519499 (+ 21580us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.519558 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.521783 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.522939 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.522954 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.523518 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:22.524353 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.524378 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.525918 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.525922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.528064 (+ 2142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.528081 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.529442 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.529449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.529757 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.530542 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:22.530572 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:22.533193 (+ 2621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:22.537129 (+ 3936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:22.537138 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:22.537158 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:22.537731 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:22.538144 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:22.538148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:22.538151 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:22.538415 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:22.539082 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:22.539101 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:22.539526 (+ 425us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.540984 (+ 1458us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.541570 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21250,"thread_start_us":21027,"threads_started":1}
I20260928 02:55:22.544792 9463 heartbeater.cc:347] Connected to a master server at 127.27.229.254:34893
I20260928 02:55:22.545125 9463 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:22.545961 9463 heartbeater.cc:511] Master 127.27.229.254:34893 requested a full tablet report, sending...
I20260928 02:55:22.547820 8987 ts_manager.cc:194] Registered new tserver with Master: cd0be9f0af9f49f0a3f10c44a91788d0 (127.27.229.195:43631)
I20260928 02:55:22.548955 8987 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:39039
I20260928 02:55:22.556033 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:55:22.591945 9466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.571265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50130 (local address 127.27.229.254:34893)
0928 02:55:22.571489 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.571495 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.572949 (+ 1454us) server_negotiation.cc:407] Connection header received
0928 02:55:22.573272 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.573279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.573371 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.573451 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:22.576336 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.577447 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.579470 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.580180 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.580325 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.584190 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:22.584197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:22.584201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:22.584250 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:22.587642 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.588622 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.588628 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.588632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.588799 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:22.589314 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:22.589321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:22.589324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:22.589725 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:22.589974 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.591332 (+ 1358us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.591595 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 02:55:22 dist-test-slave-s1kg krb5kdc[7976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564121, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for HTTP/127.27.229.212@KRBTEST.COM
I20260928 02:55:22.598865 8987 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:55:22.609118 8987 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:55:22.702877 9480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.661091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43631 (local address 127.0.0.1:58420)
0928 02:55:22.691378 (+ 30287us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.691405 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.691572 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.694296 (+ 2724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.694311 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.694365 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:22.695083 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.695102 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.697442 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.697447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.699724 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.699738 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.701758 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.701764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.702020 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.702089 (+ 69us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.702199 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.702408 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30122,"mutex_wait_us":1,"thread_start_us":29951,"threads_started":1}
I20260928 02:55:22.704815 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.661818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.27.229.195:43631)
0928 02:55:22.693566 (+ 31748us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.693575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.693610 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:55:22.693714 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.693722 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.693962 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.694179 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:22.695397 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.697282 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.699943 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.701482 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.703793 (+ 2311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.704007 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.704083 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.704262 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31567,"thread_start_us":31407,"threads_started":1}
I20260928 02:55:22.709151 9483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.660886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:37939 (local address 127.0.0.1:48138)
0928 02:55:22.685950 (+ 25064us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.686001 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.686234 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.694507 (+ 8273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.694513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.694545 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:22.695026 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.695042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.699866 (+ 4824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.699871 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.706042 (+ 6171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.706056 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.708236 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.708242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.708412 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.708450 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.708543 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.708699 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24727,"mutex_wait_us":4086,"thread_start_us":35766,"threads_started":1}
I20260928 02:55:22.709110 9398 tablet_service.cc:1467] Processing CreateTablet for tablet 28afce09fc3d4addbba5e26119fd09bd (DEFAULT_TABLE table=test-table [id=746815ce2a4249c58a9f4905126b78aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:22.710229 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.661122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48138 (local address 127.27.229.193:37939)
0928 02:55:22.692137 (+ 31015us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.692147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.692183 (+ 36us) server_negotiation.cc:407] Connection header received
0928 02:55:22.692298 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.692308 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.692612 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.692866 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:22.695465 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.699664 (+ 4199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.706207 (+ 6543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.707705 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.707903 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.708080 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.709608 (+ 1528us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.709806 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30820,"thread_start_us":34058,"threads_started":1}
I20260928 02:55:22.711124 9398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28afce09fc3d4addbba5e26119fd09bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:22.713668 9479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.672399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:33501 (local address 127.0.0.1:36894)
0928 02:55:22.696972 (+ 24573us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.696997 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.697239 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.698210 (+ 971us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.698218 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.698245 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:22.698703 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.698718 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.703806 (+ 5088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.703811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.705840 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.705857 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.712835 (+ 6978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.712842 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.713033 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.713070 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.713159 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.713325 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24430,"thread_start_us":13044,"threads_started":1}
I20260928 02:55:22.714500 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.672623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.27.229.194:33501)
0928 02:55:22.685043 (+ 12420us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.685055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.697117 (+ 12062us) server_negotiation.cc:407] Connection header received
0928 02:55:22.697440 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.697448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.697728 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.697912 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:22.702004 (+ 4092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.703591 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.710656 (+ 7065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.712271 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.712460 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.712645 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.713990 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.714179 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12209,"thread_start_us":12436,"threads_started":1}
I20260928 02:55:22.718676 9120 tablet_service.cc:1467] Processing CreateTablet for tablet 28afce09fc3d4addbba5e26119fd09bd (DEFAULT_TABLE table=test-table [id=746815ce2a4249c58a9f4905126b78aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:22.718688 9264 tablet_service.cc:1467] Processing CreateTablet for tablet 28afce09fc3d4addbba5e26119fd09bd (DEFAULT_TABLE table=test-table [id=746815ce2a4249c58a9f4905126b78aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:22.720346 9264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28afce09fc3d4addbba5e26119fd09bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:22.720376 9120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28afce09fc3d4addbba5e26119fd09bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:22.750118 9485 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Bootstrap starting.
I20260928 02:55:22.754253 9485 tablet_bootstrap.cc:654] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:22.759694 9486 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Bootstrap starting.
I20260928 02:55:22.760604 9487 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Bootstrap starting.
I20260928 02:55:22.768323 9485 log.cc:824] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:22.768621 9486 tablet_bootstrap.cc:654] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:22.770072 9486 log.cc:824] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:22.775617 9487 tablet_bootstrap.cc:654] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:22.777159 9486 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: No bootstrap required, opened a new log
I20260928 02:55:22.777560 9486 ts_tablet_manager.cc:1397] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.011s
I20260928 02:55:22.787376 9487 log.cc:824] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:22.798754 9487 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: No bootstrap required, opened a new log
I20260928 02:55:22.798763 9485 tablet_bootstrap.cc:492] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: No bootstrap required, opened a new log
I20260928 02:55:22.799126 9485 ts_tablet_manager.cc:1397] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Time spent bootstrapping tablet: real 0.049s user 0.014s sys 0.021s
I20260928 02:55:22.799257 9487 ts_tablet_manager.cc:1397] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.005s
I20260928 02:55:22.812242 9486 raft_consensus.cc:348] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.812814 9486 raft_consensus.cc:374] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:22.812927 9486 raft_consensus.cc:729] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0b5d56faf4a452aba5ed26d4a6e7da4, State: Initialized, Role: FOLLOWER
I20260928 02:55:22.813545 9487 raft_consensus.cc:348] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.814064 9487 raft_consensus.cc:374] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:22.814210 9487 raft_consensus.cc:729] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ada60dc825be49369b058b83772b08d6, State: Initialized, Role: FOLLOWER
I20260928 02:55:22.815107 9487 consensus_queue.cc:260] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.820454 9485 raft_consensus.cc:348] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.820955 9485 raft_consensus.cc:374] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:22.821066 9485 raft_consensus.cc:729] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd0be9f0af9f49f0a3f10c44a91788d0, State: Initialized, Role: FOLLOWER
I20260928 02:55:22.821791 9485 consensus_queue.cc:260] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.822698 9486 consensus_queue.cc:260] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.825356 9487 ts_tablet_manager.cc:1428] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Time spent starting tablet: real 0.026s user 0.012s sys 0.007s
I20260928 02:55:22.825804 9329 heartbeater.cc:503] Master 127.27.229.254:34893 was elected leader, sending a full tablet report...
I20260928 02:55:22.833137 9463 heartbeater.cc:503] Master 127.27.229.254:34893 was elected leader, sending a full tablet report...
I20260928 02:55:22.833943 9485 ts_tablet_manager.cc:1428] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Time spent starting tablet: real 0.035s user 0.010s sys 0.007s
I20260928 02:55:22.839186 9185 heartbeater.cc:503] Master 127.27.229.254:34893 was elected leader, sending a full tablet report...
I20260928 02:55:22.839991 9486 ts_tablet_manager.cc:1428] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Time spent starting tablet: real 0.062s user 0.019s sys 0.004s
I20260928 02:55:22.891012 9493 raft_consensus.cc:482] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:22.891360 9493 raft_consensus.cc:504] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.893546 9493 leader_election.cc:288] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd0be9f0af9f49f0a3f10c44a91788d0 (127.27.229.195:43631), ada60dc825be49369b058b83772b08d6 (127.27.229.194:33501)
W20260928 02:55:22.902653 9186 tablet.cc:2369] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:22.908764 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.894467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:33957 (local address 127.27.229.195:43631)
0928 02:55:22.894707 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.894715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.901103 (+ 6388us) server_negotiation.cc:407] Connection header received
0928 02:55:22.901349 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.901356 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.901438 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.901527 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:22.902338 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.903469 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.906507 (+ 3038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.907555 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.907675 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.907730 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.908369 (+ 639us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.908552 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:55:22.909564 9496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.893973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43631 (local address 127.27.229.193:33957)
0928 02:55:22.900921 (+ 6948us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.900952 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.901181 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.901718 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.901723 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.901750 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:22.902150 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.902166 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.903734 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.903740 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.906296 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.906315 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.907897 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.907905 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.908159 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.908194 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.909056 (+ 862us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.909208 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6795,"mutex_wait_us":70,"thread_start_us":6596,"threads_started":1}
I20260928 02:55:22.910679 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28afce09fc3d4addbba5e26119fd09bd" candidate_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" is_pre_election: true
I20260928 02:55:22.911603 9418 raft_consensus.cc:2454] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b5d56faf4a452aba5ed26d4a6e7da4 in term 0.
I20260928 02:55:22.912352 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.894685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39977 (local address 127.27.229.194:33501)
0928 02:55:22.894942 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.894951 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.904531 (+ 9580us) server_negotiation.cc:407] Connection header received
0928 02:55:22.904829 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.904835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.904911 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.904969 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:22.905827 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.906967 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.909769 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.911066 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.911202 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.911276 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.911918 (+ 642us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.912071 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:55:22.913210 9074 leader_election.cc:302] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b5d56faf4a452aba5ed26d4a6e7da4, cd0be9f0af9f49f0a3f10c44a91788d0; no voters:
I20260928 02:55:22.913748 9493 raft_consensus.cc:2781] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:55:22.913772 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.894187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:33501 (local address 127.27.229.193:39977)
0928 02:55:22.904416 (+ 10229us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:22.904437 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:22.904654 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:22.905170 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:22.905175 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:22.905200 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:22.905627 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.905640 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.907156 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.907161 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.909526 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:22.909546 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.911512 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.911518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:22.911709 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.911740 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 02:55:22.912908 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0928 02:55:22.913068 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10078,"mutex_wait_us":38,"thread_start_us":9845,"threads_started":1}
I20260928 02:55:22.913940 9493 raft_consensus.cc:482] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:55:22.914083 9493 raft_consensus.cc:3037] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:22.915153 9284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28afce09fc3d4addbba5e26119fd09bd" candidate_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada60dc825be49369b058b83772b08d6" is_pre_election: true
I20260928 02:55:22.915954 9284 raft_consensus.cc:2454] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a0b5d56faf4a452aba5ed26d4a6e7da4 in term 0.
I20260928 02:55:22.917079 9493 raft_consensus.cc:504] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.917744 9493 leader_election.cc:288] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [CANDIDATE]: Term 1 election: Requested vote from peers cd0be9f0af9f49f0a3f10c44a91788d0 (127.27.229.195:43631), ada60dc825be49369b058b83772b08d6 (127.27.229.194:33501)
I20260928 02:55:22.918239 9418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28afce09fc3d4addbba5e26119fd09bd" candidate_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0"
I20260928 02:55:22.918329 9284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28afce09fc3d4addbba5e26119fd09bd" candidate_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ada60dc825be49369b058b83772b08d6"
I20260928 02:55:22.918473 9418 raft_consensus.cc:3037] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:22.918519 9284 raft_consensus.cc:3037] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:22.921765 9284 raft_consensus.cc:2454] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b5d56faf4a452aba5ed26d4a6e7da4 in term 1.
I20260928 02:55:22.921765 9418 raft_consensus.cc:2454] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a0b5d56faf4a452aba5ed26d4a6e7da4 in term 1.
I20260928 02:55:22.922318 9075 leader_election.cc:302] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0b5d56faf4a452aba5ed26d4a6e7da4, ada60dc825be49369b058b83772b08d6; no voters:
I20260928 02:55:22.922940 9493 raft_consensus.cc:2781] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:22.923511 9493 raft_consensus.cc:686] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 1 LEADER]: Becoming Leader. State: Replica: a0b5d56faf4a452aba5ed26d4a6e7da4, State: Running, Role: LEADER
I20260928 02:55:22.924115 9493 consensus_queue.cc:237] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } }
I20260928 02:55:22.929832 8986 catalog_manager.cc:5495] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 reported cstate change: term changed from 0 to 1, leader changed from <none> to a0b5d56faf4a452aba5ed26d4a6e7da4 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0b5d56faf4a452aba5ed26d4a6e7da4" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 37939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 } health_report { overall_health: UNKNOWN } } }
I20260928 02:55:22.973122 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:22.967506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48150 (local address 127.27.229.193:37939)
0928 02:55:22.967757 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:22.967763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:22.967875 (+ 112us) server_negotiation.cc:407] Connection header received
0928 02:55:22.968162 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:22.968166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:22.968254 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:22.968409 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:22.968787 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.969546 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.970963 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:22.971367 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:22.971506 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:22.972008 (+ 502us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:22.972526 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:22.972681 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:22.972762 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 02:55:22.972845 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:55:22.981612 9284 raft_consensus.cc:1261] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 1 FOLLOWER]: Refusing update from remote peer a0b5d56faf4a452aba5ed26d4a6e7da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:22.981612 9418 raft_consensus.cc:1261] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 1 FOLLOWER]: Refusing update from remote peer a0b5d56faf4a452aba5ed26d4a6e7da4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:22.982678 9503 consensus_queue.cc:1035] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd0be9f0af9f49f0a3f10c44a91788d0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43631 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:22.983121 9493 consensus_queue.cc:1035] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ada60dc825be49369b058b83772b08d6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 33501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:22.994920 9506 mvcc.cc:205] Tried to move back new op lower bound from 7334150647722958848 to 7334150647501512704. Current Snapshot: MvccSnapshot[applied={T|T < 7334150647722958848}]
I20260928 02:55:22.997061 9507 mvcc.cc:205] Tried to move back new op lower bound from 7334150647722958848 to 7334150647501512704. Current Snapshot: MvccSnapshot[applied={T|T < 7334150647722958848}]
I20260928 02:55:23.010512 8987 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:23.010877 8987 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:23.015100 8987 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 07e2e38bdcea4752be24b6228794b924: Sending DeleteTablet for 3 replicas of tablet 28afce09fc3d4addbba5e26119fd09bd
I20260928 02:55:23.016687 9264 tablet_service.cc:1514] Processing DeleteTablet for tablet 28afce09fc3d4addbba5e26119fd09bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:36894
I20260928 02:55:23.016633 9398 tablet_service.cc:1514] Processing DeleteTablet for tablet 28afce09fc3d4addbba5e26119fd09bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:58420
I20260928 02:55:23.016745 9120 tablet_service.cc:1514] Processing DeleteTablet for tablet 28afce09fc3d4addbba5e26119fd09bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:48138
I20260928 02:55:23.027671 9516 tablet_replica.cc:330] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: stopping tablet replica
I20260928 02:55:23.028264 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9024
I20260928 02:55:23.028306 9516 raft_consensus.cc:2229] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:23.028992 9516 raft_consensus.cc:2258] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:23.030077 9518 tablet_replica.cc:330] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: stopping tablet replica
I20260928 02:55:23.030646 9518 raft_consensus.cc:2229] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:55:23.031625 9518 raft_consensus.cc:2258] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:23.034396 9516 ts_tablet_manager.cc:1905] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:23.034189 9517 tablet_replica.cc:330] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: stopping tablet replica
I20260928 02:55:23.039687 9517 raft_consensus.cc:2229] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:23.040323 9517 raft_consensus.cc:2258] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:23.042569 9517 ts_tablet_manager.cc:1905] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:23.042584 9518 ts_tablet_manager.cc:1905] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:23.047510 9516 ts_tablet_manager.cc:1918] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:23.047691 9516 log.cc:1196] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/28afce09fc3d4addbba5e26119fd09bd
I20260928 02:55:23.048230 9516 ts_tablet_manager.cc:1939] T 28afce09fc3d4addbba5e26119fd09bd P cd0be9f0af9f49f0a3f10c44a91788d0: Deleting consensus metadata
I20260928 02:55:23.049793 9517 ts_tablet_manager.cc:1918] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:23.049968 9517 log.cc:1196] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/28afce09fc3d4addbba5e26119fd09bd
I20260928 02:55:23.050211 8973 catalog_manager.cc:4841] TS cd0be9f0af9f49f0a3f10c44a91788d0 (127.27.229.195:43631): tablet 28afce09fc3d4addbba5e26119fd09bd (table test-table [id=746815ce2a4249c58a9f4905126b78aa]) successfully deleted
I20260928 02:55:23.050531 9517 ts_tablet_manager.cc:1939] T 28afce09fc3d4addbba5e26119fd09bd P ada60dc825be49369b058b83772b08d6: Deleting consensus metadata
I20260928 02:55:23.052317 8974 catalog_manager.cc:4841] TS ada60dc825be49369b058b83772b08d6 (127.27.229.194:33501): tablet 28afce09fc3d4addbba5e26119fd09bd (table test-table [id=746815ce2a4249c58a9f4905126b78aa]) successfully deleted
I20260928 02:55:23.052774 9518 ts_tablet_manager.cc:1918] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:23.052981 9518 log.cc:1196] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/28afce09fc3d4addbba5e26119fd09bd
I20260928 02:55:23.053102 9481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.029672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.27.229.193:37939)
0928 02:55:23.029874 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.029880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.029903 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:55:23.030147 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.030152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.030268 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.030373 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.033703 (+ 3330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.034974 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.037340 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.038007 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.038149 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.042579 (+ 4430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.042594 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.042611 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.042687 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.048446 (+ 5759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.049817 (+ 1371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.049827 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.049830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.050137 (+ 307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.050593 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.050597 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.050600 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.051102 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.051455 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.052401 (+ 946us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.052673 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:55:23.053902 9180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:23.053566 9518 ts_tablet_manager.cc:1939] T 28afce09fc3d4addbba5e26119fd09bd P a0b5d56faf4a452aba5ed26d4a6e7da4: Deleting consensus metadata
I20260928 02:55:23.155009 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9024
I20260928 02:55:23.155263 8972 catalog_manager.cc:4841] TS a0b5d56faf4a452aba5ed26d4a6e7da4 (127.27.229.193:37939): tablet 28afce09fc3d4addbba5e26119fd09bd (table test-table [id=746815ce2a4249c58a9f4905126b78aa]) successfully deleted
I20260928 02:55:23.172576 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9189
I20260928 02:55:23.186859 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.173595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36910 (local address 127.27.229.194:33501)
0928 02:55:23.173792 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.173798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.173820 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:55:23.173916 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.173928 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.173993 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.174104 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.175588 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.176347 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.177664 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.178068 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.178209 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.180865 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.180879 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.180896 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.180938 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.183939 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.184681 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.184698 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.184701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.184835 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.185216 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.185221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.185224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.185587 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.185918 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.186201 (+ 283us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.186457 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:55:23.187634 9324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:23.282383 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9189
I20260928 02:55:23.299269 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9333
I20260928 02:55:23.315467 9482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:23.300118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58436 (local address 127.27.229.195:43631)
0928 02:55:23.300321 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:23.300329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:23.300430 (+ 101us) server_negotiation.cc:407] Connection header received
0928 02:55:23.300655 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:23.300659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:23.300725 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:23.300844 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:23.302301 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.303906 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.305681 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:23.306095 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:23.306261 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:23.309053 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:23.309068 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:23.309084 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:23.309121 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:23.312512 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.313294 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.313302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.313304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.313426 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:23.313821 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:23.313830 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:23.313832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:23.314191 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:23.314501 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:23.314818 (+ 317us) server_negotiation.cc:299] Negotiation successful
0928 02:55:23.315067 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:55:23.316448 9458 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:23.410409 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9333
I20260928 02:55:23.427942 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 8954
I20260928 02:55:23.429196 9017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:23.543584 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 8954
I20260928 02:55:23.570487 28567 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 02:55:23.907034 28567 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 02:55:23.907229 28567 mini_ranger.cc:66] Stopping Ranger...
I20260928 02:55:24.253288 28567 mini_ranger.cc:68] Stopped Ranger
2026-09-28 02:55:24.253 UTC [8006] LOG: received smart shutdown request
2026-09-28 02:55:24.256 UTC [8006] LOG: background worker "logical replication launcher" (PID 8015) exited with exit code 1
2026-09-28 02:55:24.256 UTC [8010] LOG: shutting down
2026-09-28 02:55:24.470 UTC [8006] LOG: database system is shut down
2026-09-28T02:55:24Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (54570 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:24 dist-test-slave-s1kg krb5kdc[9531](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:24 dist-test-slave-s1kg krb5kdc[9531](info): set up 2 sockets
Sep 28 02:55:24 dist-test-slave-s1kg krb5kdc[9531](info): commencing operation
krb5kdc: starting...
W20260928 02:55:25.727370 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:55:25 dist-test-slave-s1kg krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:55:25.967172 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:40067
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:26.310263 9544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:26.310631 9544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:26.310793 9544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:26.318333 9544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:26.318497 9544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:26.318605 9544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:26.318697 9544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:26.318781 9544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:26.329213 9544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:40067
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:26.330631 9544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:26.332075 9544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:26.344873 9549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:26.344907 9552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:26.344846 9550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:26.346407 9544 server_base.cc:1034] running on GCE node
I20260928 02:55:26.347398 9544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:26.348896 9544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:26.350107 9544 hybrid_clock.cc:648] HybridClock initialized: now 1790564126350065 us; error 33 us; skew 500 ppm
I20260928 02:55:26.350478 9544 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:26 dist-test-slave-s1kg krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:26.359730 9544 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:26.361402 9544 webserver.cc:460] Webserver started at http://127.27.229.254:41665/ using document root <none> and password file <none>
I20260928 02:55:26.362154 9544 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:26.362303 9544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:26.362636 9544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:26.365862 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "65d911ff8ad04b39b937c6d17feff58f"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "7a2941cc8021c345252b07108e20f6fb"
server_key_iv: "b20690d97b1fd580220563cbbc5c2bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.366747 9544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "65d911ff8ad04b39b937c6d17feff58f"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "7a2941cc8021c345252b07108e20f6fb"
server_key_iv: "b20690d97b1fd580220563cbbc5c2bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.372746 9544 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 02:55:26.376624 9559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:26.378240 9544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:55:26.378400 9544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "65d911ff8ad04b39b937c6d17feff58f"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "7a2941cc8021c345252b07108e20f6fb"
server_key_iv: "b20690d97b1fd580220563cbbc5c2bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.378664 9544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:26.399936 9544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:26.404983 9544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:26.405316 9544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:26.426698 9544 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:40067
I20260928 02:55:26.426724 9610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:40067 every 8 connection(s)
I20260928 02:55:26.428257 9544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:26.431785 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9544
I20260928 02:55:26.432019 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:55:26.432289 9611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:26.432731 28567 external_mini_cluster.cc:1428] Setting key 50036be6aa0be96f0f012d3aa40adcd1
I20260928 02:55:26.439810 9611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.229.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.253@KRBTEST.COM" created.
Sep 28 02:55:26 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:26.469225 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.443663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40067 (local address 127.0.0.1:38934)
0928 02:55:26.444605 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.444659 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:26.446889 (+ 2230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:26.447897 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:26.447902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:26.448474 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:26.448831 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.448847 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.451286 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.451290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.452480 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.452492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.454087 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.454098 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.454410 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.455428 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:26.455489 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:26.460855 (+ 5366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:26.464430 (+ 3575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.464446 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.464463 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.465070 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.465526 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.465533 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.465540 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.465752 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.466624 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:26.466634 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:26.466911 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 02:55:26.467174 (+ 263us) client_negotiation.cc:240] Negotiation successful
0928 02:55:26.467867 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":186,"thread_start_us":436,"threads_started":1}
I20260928 02:55:26.469255 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.445107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38934 (local address 127.27.229.254:40067)
0928 02:55:26.445891 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:26.445901 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:26.445956 (+ 55us) server_negotiation.cc:407] Connection header received
0928 02:55:26.447296 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:26.447307 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:26.447621 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:26.447852 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:26.449042 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.451112 (+ 2070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.452645 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.453614 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.453837 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.461183 (+ 7346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:26.461193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:26.461207 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:26.461242 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:26.464274 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.465237 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.465243 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.465246 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.465403 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.465862 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.465865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.465867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.466410 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:26.466708 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:26.467302 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 02:55:26.467763 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":528,"thread_start_us":349,"threads_started":1}
I20260928 02:55:26.472697 9611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:26.473928 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.473508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50486)
0928 02:55:26.473693 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.473810 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 02:55:26.474360 9560 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
I20260928 02:55:26.476147 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.475696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50496)
0928 02:55:26.475869 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.475943 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260928 02:55:26.476444 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:46703: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
I20260928 02:55:26.490589 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:46703
--webserver_interface=127.27.229.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:26.525795 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } attempt: 1
I20260928 02:55:26.527055 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.526528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50506)
0928 02:55:26.526725 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.526921 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:55:26.528081 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.527726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50508)
0928 02:55:26.527878 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.527956 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:55:26.528282 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:46703: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
I20260928 02:55:26.596593 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } attempt: 2
I20260928 02:55:26.597913 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.597408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50518)
0928 02:55:26.597676 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.597768 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:55:26.598989 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.598574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50526)
0928 02:55:26.598732 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.598805 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:55:26.599179 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:46703: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
I20260928 02:55:26.690454 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } attempt: 3
I20260928 02:55:26.691740 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.691266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50538)
0928 02:55:26.691487 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.691605 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:55:26.692840 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.692460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50552)
0928 02:55:26.692621 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.692701 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:55:26.693166 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:46703: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
W20260928 02:55:26.823774 9618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:26.824076 9618 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:26.824254 9618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:26.831614 9618 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:26.831730 9618 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:26.831851 9618 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:26.831957 9618 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:26.832211 9618 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:26.836501 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } attempt: 4
I20260928 02:55:26.837805 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.837260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50556)
0928 02:55:26.837581 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.837670 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260928 02:55:26.838971 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.838555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50566)
0928 02:55:26.838734 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.838853 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260928 02:55:26.839282 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:46703: Network error: Client connection negotiation failed: client connection to 127.27.229.253:46703: connect: Connection refused (error 111)
I20260928 02:55:26.844434 9618 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:46703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.229.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:26.845801 9618 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:26.847230 9618 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:26.860345 9624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:26.860498 9626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:26.860347 9623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:26.861028 9618 server_base.cc:1034] running on GCE node
I20260928 02:55:26.861917 9618 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:26.863453 9618 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:26.864698 9618 hybrid_clock.cc:648] HybridClock initialized: now 1790564126864655 us; error 41 us; skew 500 ppm
I20260928 02:55:26.865046 9618 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:26 dist-test-slave-s1kg krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:26.868305 9618 init.cc:377] Logged in from keytab as kudu/127.27.229.253@KRBTEST.COM (short username kudu)
I20260928 02:55:26.869809 9618 webserver.cc:460] Webserver started at http://127.27.229.253:44913/ using document root <none> and password file <none>
I20260928 02:55:26.870508 9618 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:26.870663 9618 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:26.871058 9618 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:26.874078 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data/instance:
uuid: "0fa217dad04544f5b971ed64f5e911c7"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "a752383f09016f3413b749acc7b2c6e8"
server_key_iv: "9d3acd33fa18bc8c01a327546f8b9025"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.874976 9618 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal/instance:
uuid: "0fa217dad04544f5b971ed64f5e911c7"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "a752383f09016f3413b749acc7b2c6e8"
server_key_iv: "9d3acd33fa18bc8c01a327546f8b9025"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.880442 9618 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:55:26.884549 9633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:26.886123 9618 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:55:26.886335 9618 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "0fa217dad04544f5b971ed64f5e911c7"
format_stamp: "Formatted at 2026-09-28 02:55:26 on dist-test-slave-s1kg"
server_key: "a752383f09016f3413b749acc7b2c6e8"
server_key_iv: "9d3acd33fa18bc8c01a327546f8b9025"
server_key_version: "encryptionkey@0"
I20260928 02:55:26.886564 9618 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:26.900151 9618 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:26.904656 9618 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:26.904978 9618 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:26.924594 9618 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.253:46703
I20260928 02:55:26.924638 9684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.253:46703 every 8 connection(s)
I20260928 02:55:26.926056 9618 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
I20260928 02:55:26.929893 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:26.933666 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9618
I20260928 02:55:26.933892 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-1/wal/instance
I20260928 02:55:26.934340 28567 external_mini_cluster.cc:1428] Setting key 8d781215232b451e399d6386ed98ecc2
I20260928 02:55:26.936822 9685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:26 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.252@KRBTEST.COM" created.
I20260928 02:55:26.962553 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.939857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.27.229.254:40067)
0928 02:55:26.940397 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:26.940404 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:26.941619 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 02:55:26.944267 (+ 2648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:26.944273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:26.944351 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:26.944444 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:26.946684 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.947453 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.950034 (+ 2581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.950472 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.950578 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.955580 (+ 5002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:26.955586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:26.955589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:26.955626 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:26.958532 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.959463 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.959468 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.959470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.959647 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.960340 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.960345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.960349 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.960787 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:26.960944 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:26.961819 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 02:55:26.962113 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402}
I20260928 02:55:26.963393 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.940494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40067 (local address 127.0.0.1:38942)
0928 02:55:26.941403 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.941464 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:26.943912 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:26.944720 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:26.944735 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:26.945565 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:26.946419 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.946447 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.947629 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.947638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.949847 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.949866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.950728 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.950738 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.951115 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.951931 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:26.951968 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:26.955285 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:26.958709 (+ 3424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.958717 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.958731 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.959318 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.959817 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.959825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.959829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.960178 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.960935 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:26.960941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:26.961377 (+ 436us) client_negotiation.cc:769] Sending connection context
0928 02:55:26.961647 (+ 270us) client_negotiation.cc:240] Negotiation successful
0928 02:55:26.962123 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":369,"threads_started":1}
I20260928 02:55:26.965319 9685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:26 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:55:26.983421 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.966139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50582)
0928 02:55:26.966353 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.966372 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:26.966484 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:26.967630 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:26.967636 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:26.968189 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:26.968382 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.968400 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.970485 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.970489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.971446 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:26.971458 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.972738 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.972742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:26.972910 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.973577 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:26.973596 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:26.975731 (+ 2135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:26.980706 (+ 4975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.980710 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.980713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.981071 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.981518 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:26.981521 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:26.981524 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:26.981713 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:26.982539 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:26.982542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:26.982661 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 02:55:26.982816 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:55:26.983149 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":1}
I20260928 02:55:26.983642 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.966418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50582 (local address 127.27.229.253:46703)
0928 02:55:26.967202 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:26.967210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:26.967236 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:55:26.967283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:26.967288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:26.967458 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:26.967575 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:26.968549 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.970330 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.971590 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:26.972563 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:26.972716 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:26.975966 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:26.975984 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:26.976006 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:26.976048 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:26.980538 (+ 4490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.981225 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.981237 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.981244 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.981385 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:26.981846 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:26.981854 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:26.981859 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:26.982390 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:26.982605 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:26.983006 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 02:55:26.983258 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":632,"thread_start_us":451,"threads_started":1}
I20260928 02:55:26.986269 9685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:26.987528 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.987031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36650)
0928 02:55:26.987274 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.987372 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:55:26.987813 9637 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
I20260928 02:55:26.989491 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:26.989068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36666)
0928 02:55:26.989261 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:26.989342 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 02:55:26.989717 9685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:26.990556 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:43301
--webserver_interface=127.27.229.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:27.039027 9685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 1
I20260928 02:55:27.040486 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.039819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36680)
0928 02:55:27.040089 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.040247 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 02:55:27.041846 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.041324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36692)
0928 02:55:27.041549 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.041663 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 02:55:27.042111 9685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.110394 9685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 2
I20260928 02:55:27.111634 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.111109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36700)
0928 02:55:27.111332 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.111442 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:55:27.112697 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.112293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36716)
0928 02:55:27.112462 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.112550 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260928 02:55:27.112936 9685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.138607 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } attempt: 5
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:55:27.154939 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.139487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50584)
0928 02:55:27.139741 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.139760 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.139866 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.140263 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.140266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.140784 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.140965 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.140974 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.142423 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.142427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.143398 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.143410 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.144524 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.144530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.144668 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.145378 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.145397 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.147734 (+ 2337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.152137 (+ 4403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.152142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.152145 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.152549 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.153023 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.153028 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.153031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.153316 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.154008 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.154014 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.154237 (+ 223us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.154367 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.154626 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:55:27.155313 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.139617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.27.229.253:46703)
0928 02:55:27.139827 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.139834 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.139857 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:55:27.139989 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.139995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.140110 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.140221 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.141154 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.142257 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.143562 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.144303 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.144436 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.147960 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.147967 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.147972 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.148016 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.151904 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.152691 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.152697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.152700 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.152861 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.153449 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.153454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.153457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.153846 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.154001 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.154551 (+ 550us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.154886 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:55:27.156405 9611 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:55:27.157697 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.157282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36730)
0928 02:55:27.157461 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.157542 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 02:55:27.158946 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.158371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36740)
0928 02:55:27.158554 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.158728 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:55:27.159181 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.204187 9685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 3
I20260928 02:55:27.205510 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.204935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36744)
0928 02:55:27.205189 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.205315 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260928 02:55:27.206720 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.206151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36752)
0928 02:55:27.206428 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.206543 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
W20260928 02:55:27.207088 9685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.210408 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 1
I20260928 02:55:27.211524 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.211073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36758)
0928 02:55:27.211286 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.211378 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 02:55:27.212486 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.212173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36760)
0928 02:55:27.212321 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.212391 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:55:27.212700 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.280933 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 2
I20260928 02:55:27.282241 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.281727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36762)
0928 02:55:27.281981 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.282072 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 02:55:27.283411 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.283046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36778)
0928 02:55:27.283207 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.283302 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:55:27.283617 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
W20260928 02:55:27.334981 9692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:27.335351 9692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:27.335520 9692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:27.343034 9692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:27.343191 9692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:27.343307 9692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:27.343406 9692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:27.343494 9692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:27.350347 9685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 4
I20260928 02:55:27.351651 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.351095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36788)
0928 02:55:27.351352 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.351469 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:55:27.352768 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.352399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36796)
0928 02:55:27.352584 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.352660 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 02:55:27.353107 9685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.354540 9692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:40067,127.27.229.253:46703,127.27.229.252:43301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:43301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.229.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:27.355959 9692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:27.357306 9692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:27.372215 9697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:27.372407 9700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:27.372758 9698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:27.373013 9692 server_base.cc:1034] running on GCE node
I20260928 02:55:27.373878 9692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:27.375406 9692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:27.376648 9692 hybrid_clock.cc:648] HybridClock initialized: now 1790564127376577 us; error 44 us; skew 500 ppm
I20260928 02:55:27.377051 9692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:27.380484 9692 init.cc:377] Logged in from keytab as kudu/127.27.229.252@KRBTEST.COM (short username kudu)
I20260928 02:55:27.382175 9692 webserver.cc:460] Webserver started at http://127.27.229.252:35459/ using document root <none> and password file <none>
I20260928 02:55:27.383354 9692 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:27.383538 9692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:27.383883 9692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:27.387172 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data/instance:
uuid: "b589db0c58be4e71a5d013890558ef54"
format_stamp: "Formatted at 2026-09-28 02:55:27 on dist-test-slave-s1kg"
server_key: "c02c056663e4ce179269ed9c4ca223cb"
server_key_iv: "11cab7680a057b4b604fa118b3f5b734"
server_key_version: "encryptionkey@0"
I20260928 02:55:27.388018 9692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal/instance:
uuid: "b589db0c58be4e71a5d013890558ef54"
format_stamp: "Formatted at 2026-09-28 02:55:27 on dist-test-slave-s1kg"
server_key: "c02c056663e4ce179269ed9c4ca223cb"
server_key_iv: "11cab7680a057b4b604fa118b3f5b734"
server_key_version: "encryptionkey@0"
I20260928 02:55:27.389847 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 3
I20260928 02:55:27.391227 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.390568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36806)
0928 02:55:27.390797 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.390954 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:55:27.392386 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.391930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36818)
0928 02:55:27.392139 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.392251 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 02:55:27.392655 9611 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:43301: Network error: Client connection negotiation failed: client connection to 127.27.229.252:43301: connect: Connection refused (error 111)
I20260928 02:55:27.394543 9692 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260928 02:55:27.398629 9707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:27.400241 9692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:27.400463 9692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "b589db0c58be4e71a5d013890558ef54"
format_stamp: "Formatted at 2026-09-28 02:55:27 on dist-test-slave-s1kg"
server_key: "c02c056663e4ce179269ed9c4ca223cb"
server_key_iv: "11cab7680a057b4b604fa118b3f5b734"
server_key_version: "encryptionkey@0"
I20260928 02:55:27.400687 9692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:27.437110 9692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:27.441908 9692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:27.442232 9692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:27.461494 9692 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.252:43301
I20260928 02:55:27.461542 9758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.252:43301 every 8 connection(s)
I20260928 02:55:27.463023 9692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
I20260928 02:55:27.464288 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9692
I20260928 02:55:27.464489 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790563915649057-28567-0/minicluster-data/master-2/wal/instance
I20260928 02:55:27.465054 28567 external_mini_cluster.cc:1428] Setting key ea062f4c49cee43db843c7b6668809e1
I20260928 02:55:27.467958 9759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:27.476518 9759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:27.491389 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.467276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.27.229.252:43301)
0928 02:55:27.468173 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.468188 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.468245 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:55:27.469866 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.469888 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.470278 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.470688 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.472559 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.473979 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.475945 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.476635 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.476778 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.481959 (+ 5181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.481971 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.481985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.482041 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.486199 (+ 4158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.487228 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.487239 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.487250 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.487482 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.487875 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.487878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.487880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.488697 (+ 817us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.489010 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.489469 (+ 459us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.489930 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":420,"threads_started":1}
I20260928 02:55:27.498111 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.479846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38948 (local address 127.27.229.254:40067)
0928 02:55:27.480402 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.480409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.481001 (+ 592us) server_negotiation.cc:407] Connection header received
0928 02:55:27.481210 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.481215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.481305 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.481435 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.483056 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.484210 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.486366 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.487096 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.487226 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.491146 (+ 3920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.491152 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.491156 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.491194 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.494725 (+ 3531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.495695 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.495704 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.495708 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.495902 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.496308 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.496314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.496318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.496745 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.496938 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.497540 (+ 602us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.497745 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":297,"threads_started":1}
I20260928 02:55:27.498229 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.479844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:40067 (local address 127.0.0.1:38948)
0928 02:55:27.480844 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.480888 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.481061 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.481504 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.481509 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.482414 (+ 905us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.482882 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.482903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.484381 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.484389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.486168 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.486183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.487311 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.487322 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.487502 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.488136 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.488158 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.490896 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.494932 (+ 4036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.494946 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.494956 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.495557 (+ 601us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.496036 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.496040 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.496042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.496183 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.496917 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.496923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.497079 (+ 156us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.497393 (+ 314us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.497756 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":590,"threads_started":1}
I20260928 02:55:27.499833 9759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:55:27.512715 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.494500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.27.229.254:40067)
0928 02:55:27.495248 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.495254 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.495278 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:55:27.495331 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.495337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.495414 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.495530 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.497385 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.498748 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.500645 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.501205 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.501307 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.506119 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.506127 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.506133 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.506177 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.509595 (+ 3418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.510562 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.510567 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.510569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.510697 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.511150 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.511154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.511156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.511485 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.511780 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.512106 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.512352 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":400,"threads_started":1}
I20260928 02:55:27.514189 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.500816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.27.229.253:46703)
0928 02:55:27.501082 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.501108 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.501132 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:55:27.501198 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.501202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.501289 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.501397 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.502087 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.502743 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.503950 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.504398 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.504542 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.508156 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.508168 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.508171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.508204 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.511355 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.512014 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.512026 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.512029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.512171 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.512604 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.512608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.512612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.512938 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.513088 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.513540 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.513772 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:55:27.514407 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.500712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:46703 (local address 127.0.0.1:50592)
0928 02:55:27.500914 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.500935 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.501040 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.501448 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.501455 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.501793 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.501935 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.501942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.502905 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.502912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.503793 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.503804 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.504563 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.504568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.504818 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.505471 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.505490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.507939 (+ 2449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.511487 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.511492 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.511496 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.511879 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.512287 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.512292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.512296 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.512436 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.513067 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.513072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.513254 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.513424 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.513970 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:55:27.515553 9759 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564127, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:55:27.530360 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.516406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.27.229.252:43301)
0928 02:55:27.516604 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.516611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.516632 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:55:27.516687 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.516692 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.516776 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.516889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.517691 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.518585 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.519870 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.520378 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.520517 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.523712 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.523723 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.523729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.523775 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.527430 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.528125 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.528134 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.528142 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.528301 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.528678 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.528686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.528691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.529071 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.529237 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.529634 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.529887 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:55:27.530472 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.516320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36838)
0928 02:55:27.516467 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.516486 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.516602 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.516921 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.516924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.517333 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.517512 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.517559 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.518739 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.518742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.519729 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.519742 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.520572 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.520576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.520716 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.521430 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.521447 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.523493 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.527615 (+ 4122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.527619 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.527622 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.527987 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.528447 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.528450 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.528452 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.528569 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.529211 (+ 642us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.529217 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.529396 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.529555 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.530056 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":162}
I20260928 02:55:27.531867 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.515023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50602 (local address 127.27.229.253:46703)
0928 02:55:27.515291 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.515296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.515313 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:55:27.515390 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.515395 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.515467 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.515557 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.517265 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.518136 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.519725 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.520205 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.520296 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.525201 (+ 4905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.525208 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.525214 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.525261 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.528777 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.529510 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.529514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.529516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.529649 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.530049 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.530053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.530055 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.530316 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.531149 (+ 833us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.531295 (+ 146us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.531528 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260928 02:55:27.536391 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: Bootstrap starting.
I20260928 02:55:27.536835 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:27.540884 9759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:27.542685 9759 log.cc:824] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:27.546802 9759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: No bootstrap required, opened a new log
I20260928 02:55:27.558310 9759 raft_consensus.cc:348] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.558877 9759 raft_consensus.cc:374] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:27.559020 9759 raft_consensus.cc:729] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b589db0c58be4e71a5d013890558ef54, State: Initialized, Role: FOLLOWER
I20260928 02:55:27.559906 9759 consensus_queue.cc:260] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.567181 9774 sys_catalog.cc:455] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.567727 9765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.544371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38974 (local address 127.27.229.254:40067)
0928 02:55:27.544676 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.544686 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.545144 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:55:27.545481 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.545488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.545593 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.545716 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.548455 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.549686 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.552566 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.553498 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.553627 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.559456 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.559463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.559468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.559511 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.563360 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.564230 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.564238 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.564242 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.564379 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.565788 (+ 1409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.565796 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.565801 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.566226 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.566482 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.567110 (+ 628us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.567366 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:55:27.567716 9774 sys_catalog.cc:458] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.569880 9611 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 4
I20260928 02:55:27.570396 9759 sys_catalog.cc:564] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:27.573326 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.544377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50616 (local address 127.27.229.253:46703)
0928 02:55:27.544849 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.544858 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.551468 (+ 6610us) server_negotiation.cc:407] Connection header received
0928 02:55:27.551784 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.551793 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.551880 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.551955 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.554926 (+ 2971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.556169 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.559264 (+ 3095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.559927 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.560036 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.563966 (+ 3930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.563976 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.564003 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.564054 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.567588 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.569805 (+ 2217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.569810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.569813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.569978 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.570523 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.570528 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.570533 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.571029 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.571300 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.572457 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.572809 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260928 02:55:27.580207 9762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.545274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.27.229.252:43301)
0928 02:55:27.545443 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.545451 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.559217 (+ 13766us) server_negotiation.cc:407] Connection header received
0928 02:55:27.559515 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.559521 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.559620 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.559700 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.562163 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.563790 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.566777 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.567679 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.567798 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.571615 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.571623 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.571629 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.571676 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.576219 (+ 4543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.577185 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.577198 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.577205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.577427 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.577903 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.577911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.577914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.578415 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.578680 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.579288 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.579567 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:55:27.590315 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.570503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36854)
0928 02:55:27.571506 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.571531 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.571671 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.575454 (+ 3783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.575464 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.576181 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.576507 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.576524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.578185 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.578191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.579614 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.579632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.580467 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.580473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.580655 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.581377 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.581399 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.583791 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.587716 (+ 3925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.587722 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.587731 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.588135 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.588522 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.588527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.588532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.588674 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.589201 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.589207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.589361 (+ 154us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.589531 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.589847 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831}
I20260928 02:55:27.591125 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.570856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36854 (local address 127.27.229.252:43301)
0928 02:55:27.574935 (+ 4079us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.574945 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.574980 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:55:27.575055 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.575063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.575174 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.576402 (+ 1228us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.576698 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.578010 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.579793 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.580259 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.580946 (+ 687us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.584017 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.584022 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.584026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.584054 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.587508 (+ 3454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.588264 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.588269 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.588272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.588408 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.588792 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.588801 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.588803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.589083 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.589233 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.589851 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.590159 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3584,"thread_start_us":3386,"threads_started":1}
I20260928 02:55:27.598392 9611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: Bootstrap starting.
W20260928 02:55:27.596433 9786 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:27.599915 9786 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:27.602805 9611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:27.604245 9611 log.cc:824] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:27.593101 9692 master_runner.cc:186] Error getting master registration for 127.27.229.254:40067: OK, instance_id { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" instance_seqno: 1790564126420464 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:55:27.606160 9692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:27.607915 9611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: No bootstrap required, opened a new log
I20260928 02:55:27.616744 9611 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.617123 9611 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:27.617281 9611 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65d911ff8ad04b39b937c6d17feff58f, State: Initialized, Role: FOLLOWER
I20260928 02:55:27.617899 9611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.618856 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.619270 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.620090 9611 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: configured and running, proceeding with master startup.
W20260928 02:55:27.633518 9800 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:27.633771 9800 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:27.631402 9544 master_runner.cc:186] Error getting master registration for 127.27.229.253:46703: OK, instance_id { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" instance_seqno: 1790564126918820 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:55:27.643606 9544 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:27.652518 9685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } attempt: 5
W20260928 02:55:27.658689 9565 tablet.cc:2369] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 02:55:27 dist-test-slave-s1kg krb5kdc[9531](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564126, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:55:27.672475 9775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.653858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36864 (local address 127.27.229.252:43301)
0928 02:55:27.654033 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:27.654039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:27.654056 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:55:27.654115 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:27.654121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:27.654210 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:27.654303 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:27.655526 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.656825 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.658660 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.659425 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.659554 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.664221 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:27.664230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:27.664236 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:27.664287 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:27.669285 (+ 4998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.670147 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.670153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.670156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.670328 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:27.670800 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:27.670805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:27.670808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:27.671187 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:27.671338 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:27.671854 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 02:55:27.672093 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 02:55:27.672860 9687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:27.653172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:43301 (local address 127.0.0.1:36864)
0928 02:55:27.653526 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:27.653546 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:27.653661 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:27.654518 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:27.654523 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:27.655130 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:27.655362 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.655374 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.656990 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.656996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.658464 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:27.658481 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:27.659780 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:27.659788 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:27.660032 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:27.660785 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:27.660813 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:27.663976 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:27.669436 (+ 5460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.669442 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.669447 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.670013 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.670455 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:27.670460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:27.670465 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:27.670676 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:27.671366 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:27.671372 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:27.671546 (+ 174us) client_negotiation.cc:769] Sending connection context
0928 02:55:27.671737 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 02:55:27.672143 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260928 02:55:27.679772 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: Bootstrap starting.
I20260928 02:55:27.684688 9685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:27.686303 9685 log.cc:824] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:27.689998 9685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: No bootstrap required, opened a new log
W20260928 02:55:27.694800 9713 tablet.cc:2369] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:27.702607 9685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.703235 9685 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:27.703411 9685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa217dad04544f5b971ed64f5e911c7, State: Initialized, Role: FOLLOWER
I20260928 02:55:27.704327 9685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.705469 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.705935 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.707238 9685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:27.719024 9774 raft_consensus.cc:482] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:27.719362 9774 raft_consensus.cc:504] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.720964 9774 leader_election.cc:288] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65d911ff8ad04b39b937c6d17feff58f (127.27.229.254:40067), 0fa217dad04544f5b971ed64f5e911c7 (127.27.229.253:46703)
I20260928 02:55:27.722126 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b589db0c58be4e71a5d013890558ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa217dad04544f5b971ed64f5e911c7" is_pre_election: true
I20260928 02:55:27.722673 9660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b589db0c58be4e71a5d013890558ef54 in term 0.
I20260928 02:55:27.723315 9708 leader_election.cc:302] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fa217dad04544f5b971ed64f5e911c7, b589db0c58be4e71a5d013890558ef54; no voters:
I20260928 02:55:27.723759 9774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:55:27.723902 9774 raft_consensus.cc:482] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:55:27.724021 9774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:27.724350 9586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b589db0c58be4e71a5d013890558ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d911ff8ad04b39b937c6d17feff58f" is_pre_election: true
I20260928 02:55:27.724901 9586 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b589db0c58be4e71a5d013890558ef54 in term 0.
I20260928 02:55:27.727593 9774 raft_consensus.cc:504] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.728663 9586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b589db0c58be4e71a5d013890558ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65d911ff8ad04b39b937c6d17feff58f"
I20260928 02:55:27.728871 9586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:27.730990 9618 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260928 02:55:27.731235 9618 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:55:27.732156 9586 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b589db0c58be4e71a5d013890558ef54 in term 1.
I20260928 02:55:27.732577 9711 leader_election.cc:302] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65d911ff8ad04b39b937c6d17feff58f, b589db0c58be4e71a5d013890558ef54; no voters:
W20260928 02:55:27.735198 9814 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:55:27.735333 9814 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:55:27.735939 9774 leader_election.cc:288] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [CANDIDATE]: Term 1 election: Requested vote from peers 65d911ff8ad04b39b937c6d17feff58f (127.27.229.254:40067), 0fa217dad04544f5b971ed64f5e911c7 (127.27.229.253:46703)
I20260928 02:55:27.736481 9660 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b589db0c58be4e71a5d013890558ef54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0fa217dad04544f5b971ed64f5e911c7"
I20260928 02:55:27.736724 9660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:27.736974 9774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:27.737238 9774 raft_consensus.cc:686] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [term 1 LEADER]: Becoming Leader. State: Replica: b589db0c58be4e71a5d013890558ef54, State: Running, Role: LEADER
I20260928 02:55:27.737964 9774 consensus_queue.cc:237] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } }
I20260928 02:55:27.740144 9660 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b589db0c58be4e71a5d013890558ef54 in term 1.
I20260928 02:55:27.755118 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b589db0c58be4e71a5d013890558ef54. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.755673 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:27.767518 9818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:27.770047 9818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:27.792932 9660 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [term 1 FOLLOWER]: Refusing update from remote peer b589db0c58be4e71a5d013890558ef54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:27.794129 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:27.801772 9586 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [term 1 FOLLOWER]: Refusing update from remote peer b589db0c58be4e71a5d013890558ef54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:27.802750 9816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:27.837872 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: SysCatalogTable state changed. Reason: New leader b589db0c58be4e71a5d013890558ef54. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.838234 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.844010 9819 mvcc.cc:205] Tried to move back new op lower bound from 7334150667425415168 to 7334150667226673152. Current Snapshot: MvccSnapshot[applied={T|T < 7334150667425415168}]
I20260928 02:55:27.848155 9821 mvcc.cc:205] Tried to move back new op lower bound from 7334150667425415168 to 7334150667226673152. Current Snapshot: MvccSnapshot[applied={T|T < 7334150667425415168}]
I20260928 02:55:27.852083 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b589db0c58be4e71a5d013890558ef54. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.852412 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.860702 9820 mvcc.cc:205] Tried to move back new op lower bound from 7334150667425415168 to 7334150667226673152. Current Snapshot: MvccSnapshot[applied={T|T < 7334150667425415168}]
I20260928 02:55:27.861249 9789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.861521 9789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.861276 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.861639 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:27.862181 9816 sys_catalog.cc:455] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.862470 9816 sys_catalog.cc:458] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:27.863588 9803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b589db0c58be4e71a5d013890558ef54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65d911ff8ad04b39b937c6d17feff58f" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 40067 } } peers { permanent_uuid: "0fa217dad04544f5b971ed64f5e911c7" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 46703 } } peers { permanent_uuid: "b589db0c58be4e71a5d013890558ef54" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 43301 } } }
I20260928 02:55:27.863835 9803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:55:27.867501 9818 catalog_manager.cc:1347] Generated new cluster ID: 189d08af50254849bbfa944aa071cfe1
I20260928 02:55:27.867610 9818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:27.890702 9818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:27.892750 9818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:27.909374 9818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b589db0c58be4e71a5d013890558ef54: Generated new TSK 0
I20260928 02:55:27.910486 9818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:28.635890 9800 catalog_manager.cc:1259] Loaded cluster ID: 189d08af50254849bbfa944aa071cfe1
I20260928 02:55:28.636008 9800 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: loading cluster ID for follower catalog manager: success
I20260928 02:55:28.638523 9800 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: acquiring CA information for follower catalog manager: success
I20260928 02:55:28.640539 9800 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 65d911ff8ad04b39b937c6d17feff58f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:28.737524 9814 catalog_manager.cc:1259] Loaded cluster ID: 189d08af50254849bbfa944aa071cfe1
I20260928 02:55:28.737681 9814 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: loading cluster ID for follower catalog manager: success
I20260928 02:55:28.740456 9814 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: acquiring CA information for follower catalog manager: success
I20260928 02:55:28.742260 9814 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0fa217dad04544f5b971ed64f5e911c7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:55:29.076953 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9544
I20260928 02:55:29.078323 9606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:29.188454 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9544
I20260928 02:55:29.213901 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9618
I20260928 02:55:29.215095 9680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:29.322459 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9618
I20260928 02:55:29.342715 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9692
I20260928 02:55:29.343957 9754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:29.458868 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9692
2026-09-28T02:55:29Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4861 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:29 dist-test-slave-s1kg krb5kdc[9838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:29 dist-test-slave-s1kg krb5kdc[9838](info): set up 2 sockets
Sep 28 02:55:29 dist-test-slave-s1kg krb5kdc[9838](info): commencing operation
krb5kdc: starting...
W20260928 02:55:30.579411 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:55:30 dist-test-slave-s1kg krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:55:30.831453 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44367
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:44367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:31.171187 9851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:31.171489 9851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:31.171581 9851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:31.179010 9851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:31.179145 9851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:31.179203 9851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:31.179252 9851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:31.179297 9851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:31.189477 9851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:44367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:31.190869 9851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:31.192157 9851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:31.204927 9857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:31.204927 9856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:31.204962 9859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:31.205811 9851 server_base.cc:1034] running on GCE node
I20260928 02:55:31.206709 9851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:31.208319 9851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:31.209602 9851 hybrid_clock.cc:648] HybridClock initialized: now 1790564131209567 us; error 54 us; skew 500 ppm
I20260928 02:55:31.210105 9851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:31 dist-test-slave-s1kg krb5kdc[9838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564131, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:31.213888 9851 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:31.215626 9851 webserver.cc:460] Webserver started at http://127.27.229.254:43853/ using document root <none> and password file <none>
I20260928 02:55:31.216379 9851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:31.216537 9851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:31.216871 9851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:31.220131 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "51cdf6aea4f8412b888ab238739cbfc0"
format_stamp: "Formatted at 2026-09-28 02:55:31 on dist-test-slave-s1kg"
server_key: "b711cd16d4eb6994c38fb3b8afce0556"
server_key_iv: "11e756a0d9bb70733170f622323107c2"
server_key_version: "encryptionkey@0"
I20260928 02:55:31.220964 9851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "51cdf6aea4f8412b888ab238739cbfc0"
format_stamp: "Formatted at 2026-09-28 02:55:31 on dist-test-slave-s1kg"
server_key: "b711cd16d4eb6994c38fb3b8afce0556"
server_key_iv: "11e756a0d9bb70733170f622323107c2"
server_key_version: "encryptionkey@0"
I20260928 02:55:31.226688 9851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:55:31.230485 9866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:31.232131 9851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:31.232339 9851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "51cdf6aea4f8412b888ab238739cbfc0"
format_stamp: "Formatted at 2026-09-28 02:55:31 on dist-test-slave-s1kg"
server_key: "b711cd16d4eb6994c38fb3b8afce0556"
server_key_iv: "11e756a0d9bb70733170f622323107c2"
server_key_version: "encryptionkey@0"
I20260928 02:55:31.232558 9851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:31.252715 9851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:31.257481 9851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:31.257789 9851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:31.278195 9851 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:44367
I20260928 02:55:31.278249 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:44367 every 8 connection(s)
I20260928 02:55:31.279755 9851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:31.283643 9918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:31.285497 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9851
I20260928 02:55:31.285724 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:55:31.286389 28567 external_mini_cluster.cc:1428] Setting key 9d3be73cfec143bee9a5999285e42f7c
I20260928 02:55:31.296515 9918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0: Bootstrap starting.
I20260928 02:55:31.300658 9918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:31.302119 9918 log.cc:824] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:55:31 dist-test-slave-s1kg krb5kdc[9838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:31.305483 9918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0: No bootstrap required, opened a new log
I20260928 02:55:31.311887 9921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:31.288552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59698 (local address 127.27.229.254:44367)
0928 02:55:31.289422 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:31.289435 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:31.289483 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:55:31.291475 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:31.291499 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:31.291902 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:31.292258 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:31.294251 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:31.295797 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:31.297652 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:31.298385 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:31.298569 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:31.304079 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:31.304088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:31.304104 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:31.304141 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:31.307470 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:31.308274 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:31.308280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:31.308287 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:31.308468 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:31.308906 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:31.308913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:31.308921 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:31.309465 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:31.309819 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:31.310283 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 02:55:31.310762 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":396,"threads_started":1}
I20260928 02:55:31.313712 9918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } }
I20260928 02:55:31.314111 9918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:31.314248 9918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51cdf6aea4f8412b888ab238739cbfc0, State: Initialized, Role: FOLLOWER
I20260928 02:55:31.315032 9918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } }
I20260928 02:55:31.315335 9918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:31.315485 9918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:31.315670 9918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:31.316028 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:55:31.318998 9918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } }
I20260928 02:55:31.319660 9918 leader_election.cc:302] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51cdf6aea4f8412b888ab238739cbfc0; no voters:
I20260928 02:55:31.320386 9918 leader_election.cc:288] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:31.320536 9923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:31.321261 9923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [term 1 LEADER]: Becoming Leader. State: Replica: 51cdf6aea4f8412b888ab238739cbfc0, State: Running, Role: LEADER
I20260928 02:55:31.321808 9923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } }
I20260928 02:55:31.322570 9918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:31.325042 9925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51cdf6aea4f8412b888ab238739cbfc0. Latest consensus state: current_term: 1 leader_uuid: "51cdf6aea4f8412b888ab238739cbfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } } }
I20260928 02:55:31.325018 9924 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51cdf6aea4f8412b888ab238739cbfc0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51cdf6aea4f8412b888ab238739cbfc0" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44367 } } }
I20260928 02:55:31.325469 9925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:31.325505 9924 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:31.328339 9930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:31.334172 9930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:31.344183 9930 catalog_manager.cc:1347] Generated new cluster ID: 89ec6cf7a380448cac3aa44d30c94969
I20260928 02:55:31.344357 9930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:31.379876 9930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:31.381279 9930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:31.394364 9930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51cdf6aea4f8412b888ab238739cbfc0: Generated new TSK 0
I20260928 02:55:31.395290 9930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:55:31.430197 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9851
I20260928 02:55:31.431557 9913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:31.529238 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9851
2026-09-28T02:55:31Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2064 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:31 dist-test-slave-s1kg krb5kdc[9946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:31 dist-test-slave-s1kg krb5kdc[9946](info): set up 2 sockets
Sep 28 02:55:31 dist-test-slave-s1kg krb5kdc[9946](info): commencing operation
krb5kdc: starting...
W20260928 02:55:32.655686 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:55:32 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:55:32.907399 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:33.245440 9959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:33.245759 9959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:33.245913 9959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:33.253415 9959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:33.253546 9959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:33.253593 9959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:33.253714 9959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:33.253777 9959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:33.264379 9959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:33.265870 9959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:33.267256 9959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:33.279763 9964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:33.279832 9967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:33.280007 9965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:33.280866 9959 server_base.cc:1034] running on GCE node
I20260928 02:55:33.281617 9959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:33.283079 9959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:33.284308 9959 hybrid_clock.cc:648] HybridClock initialized: now 1790564133284262 us; error 31 us; skew 500 ppm
I20260928 02:55:33.284633 9959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:33 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:33.288002 9959 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:33.289688 9959 webserver.cc:460] Webserver started at http://127.27.229.254:37513/ using document root <none> and password file <none>
I20260928 02:55:33.290511 9959 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:33.290637 9959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:33.290972 9959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:33.294103 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "cb3cc013b5634f36bd62bc3561a04696"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "c33cbd0e678ef94bb33854303f397a7f"
server_key_iv: "a2de3bfad0e018ef3c05a93298871eda"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.294994 9959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "cb3cc013b5634f36bd62bc3561a04696"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "c33cbd0e678ef94bb33854303f397a7f"
server_key_iv: "a2de3bfad0e018ef3c05a93298871eda"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.300558 9959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 02:55:33.304343 9974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:33.305878 9959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:55:33.306100 9959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "cb3cc013b5634f36bd62bc3561a04696"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "c33cbd0e678ef94bb33854303f397a7f"
server_key_iv: "a2de3bfad0e018ef3c05a93298871eda"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.306327 9959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:33.325039 9959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:33.329285 9959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:33.329576 9959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:33.348745 9959 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:46197
I20260928 02:55:33.348780 10025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:46197 every 8 connection(s)
I20260928 02:55:33.350150 9959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:33.351477 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 9959
I20260928 02:55:33.351711 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:55:33.352510 28567 external_mini_cluster.cc:1428] Setting key e91697244da4d36199127e1a15135055
I20260928 02:55:33.355039 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:33.368990 10026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap starting.
Sep 28 02:55:33 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:33.373068 10026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:33.374475 10026 log.cc:824] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:33.378262 10026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: No bootstrap required, opened a new log
I20260928 02:55:33.379904 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:33.354705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52616 (local address 127.27.229.254:46197)
0928 02:55:33.355583 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:33.355595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:33.355635 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:55:33.357256 (+ 1621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:33.357277 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:33.357661 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:33.358075 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:33.360047 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.362090 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.364251 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.365378 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.365587 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:33.370683 (+ 5096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:33.370694 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:33.370715 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:33.370768 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:33.374810 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:33.375910 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:33.375918 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:33.375924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:33.376205 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:33.376611 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:33.376617 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:33.376624 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:33.377342 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:33.377663 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:33.378075 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 02:55:33.378519 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":447,"threads_started":1}
I20260928 02:55:33.387199 10026 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:33.387574 10026 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:33.387655 10026 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Initialized, Role: FOLLOWER
I20260928 02:55:33.388149 10026 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:33.388356 10026 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:33.388453 10026 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:33.388588 10026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:33.391012 10026 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:33.391520 10026 leader_election.cc:302] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3cc013b5634f36bd62bc3561a04696; no voters:
I20260928 02:55:33.392092 10026 leader_election.cc:288] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:33.392282 10031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:33.392988 10031 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 LEADER]: Becoming Leader. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Running, Role: LEADER
I20260928 02:55:33.393528 10031 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:33.394186 10026 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:33.396773 10032 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:33.396750 10033 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3cc013b5634f36bd62bc3561a04696. Latest consensus state: current_term: 1 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:33.397240 10032 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:33.397327 10033 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:33.400163 10037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:33.405447 10037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:33.415324 10037 catalog_manager.cc:1347] Generated new cluster ID: 12715fb200b24911865280ca0b18e6b9
I20260928 02:55:33.415478 10037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:33.428550 10037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:33.429908 10037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:33.440678 10037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Generated new TSK 0
I20260928 02:55:33.441566 10037 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:55:33.499486 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:33.838958 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:33.839267 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:33.839363 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:33.847036 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:33.847167 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:33.847283 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:33.857369 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:33.858655 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:33.859964 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:33.874312 10057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:33.874238 10060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:33.874272 10058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:33.875552 10052 server_base.cc:1034] running on GCE node
I20260928 02:55:33.876497 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:33.877585 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:33.878918 10052 hybrid_clock.cc:648] HybridClock initialized: now 1790564133878783 us; error 62 us; skew 500 ppm
I20260928 02:55:33.879326 10052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:33 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:33.883569 10052 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:33.885208 10052 webserver.cc:460] Webserver started at http://127.27.229.193:36485/ using document root <none> and password file <none>
I20260928 02:55:33.885902 10052 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:33.886008 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:33.886276 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:33.889293 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "406a33188efa422e8ec38541b4a1b812"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "d5e3b2ea865bd51be97ac31714773631"
server_key_iv: "513110edede80db64a8c132125d215e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.890038 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "406a33188efa422e8ec38541b4a1b812"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "d5e3b2ea865bd51be97ac31714773631"
server_key_iv: "513110edede80db64a8c132125d215e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.895743 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:55:33.899358 10067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:33.901151 10052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:55:33.901304 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "406a33188efa422e8ec38541b4a1b812"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "d5e3b2ea865bd51be97ac31714773631"
server_key_iv: "513110edede80db64a8c132125d215e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:33.901484 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:33.917383 10052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:33.921921 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:33.922185 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:33.923286 10052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:33.924724 10052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:33.924817 10052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:33.924916 10052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:33.924963 10052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:33.959049 10052 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39057
I20260928 02:55:33.959091 10179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39057 every 8 connection(s)
I20260928 02:55:33.960430 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:33.964561 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10052
I20260928 02:55:33.964814 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:55:33.965503 28567 external_mini_cluster.cc:1428] Setting key ffc998c0ac71ff31c350e93d3e5d1c1b
Sep 28 02:55:33 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564133, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:33.982460 10182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:33.963104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:36685 (local address 127.27.229.254:46197)
0928 02:55:33.963577 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:33.963581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:33.964567 (+ 986us) server_negotiation.cc:407] Connection header received
0928 02:55:33.966735 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:33.966740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:33.966810 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:33.966974 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:33.968661 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.969373 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.971062 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.971470 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.971616 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:33.975766 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:33.975772 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:33.975776 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:33.975821 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:33.979125 (+ 3304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:33.979949 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:33.979959 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:33.979962 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:33.980095 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:33.980565 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:33.980574 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:33.980576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:33.980849 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:33.981025 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:33.981806 (+ 781us) server_negotiation.cc:299] Negotiation successful
0928 02:55:33.982071 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":366,"thread_start_us":269,"threads_started":1}
I20260928 02:55:33.983589 10183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:33.963610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:36685)
0928 02:55:33.964397 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:33.964439 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:33.966443 (+ 2004us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:33.967265 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:33.967277 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:33.967884 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:33.968474 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:33.968491 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.969554 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.969559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:33.970895 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:33.970919 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:33.971727 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:33.971734 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:33.972020 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:33.972796 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:33.972820 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:33.975477 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:33.979305 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:33.979322 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:33.979338 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:33.979798 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:33.980220 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:33.980225 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:33.980229 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:33.980436 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:33.980994 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:33.981003 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:33.981393 (+ 390us) client_negotiation.cc:769] Sending connection context
0928 02:55:33.981689 (+ 296us) client_negotiation.cc:240] Negotiation successful
0928 02:55:33.982296 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":375,"threads_started":1}
I20260928 02:55:33.985483 10180 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
I20260928 02:55:33.985826 10180 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:33.986604 10180 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:55:33.988992 9991 ts_manager.cc:194] Registered new tserver with Master: 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057)
I20260928 02:55:33.990504 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:36685
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:55:34.021770 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:34.358676 10186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:34.359061 10186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:34.359262 10186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:34.366778 10186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:34.366976 10186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:34.367202 10186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:34.377511 10186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:34.378888 10186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:34.380206 10186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:34.394076 10191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:34.394065 10192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:34.394076 10194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:34.394886 10186 server_base.cc:1034] running on GCE node
I20260928 02:55:34.395669 10186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:34.396728 10186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:34.397956 10186 hybrid_clock.cc:648] HybridClock initialized: now 1790564134397885 us; error 58 us; skew 500 ppm
I20260928 02:55:34.398365 10186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:34 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:34.401815 10186 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:34.403502 10186 webserver.cc:460] Webserver started at http://127.27.229.194:46873/ using document root <none> and password file <none>
I20260928 02:55:34.404296 10186 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:34.404444 10186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:34.404752 10186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:34.408185 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "37468f93c61143a49e27aadf1258e984"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "85372e2533ae1f017612d7f27a827182"
server_key_iv: "c4859c89fc46d6ecc1d6319ebb717bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.409022 10186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "37468f93c61143a49e27aadf1258e984"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "85372e2533ae1f017612d7f27a827182"
server_key_iv: "c4859c89fc46d6ecc1d6319ebb717bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.414523 10186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:55:34.418080 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.419580 10186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:34.419780 10186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "37468f93c61143a49e27aadf1258e984"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "85372e2533ae1f017612d7f27a827182"
server_key_iv: "c4859c89fc46d6ecc1d6319ebb717bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.419996 10186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:34.452075 10186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:34.456472 10186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:34.456782 10186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:34.457798 10186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:34.459254 10186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:34.459357 10186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.459511 10186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:34.459599 10186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.493762 10186 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41937
I20260928 02:55:34.493850 10313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41937 every 8 connection(s)
I20260928 02:55:34.495250 10186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:34.496172 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10186
I20260928 02:55:34.496373 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:55:34.496835 28567 external_mini_cluster.cc:1428] Setting key af1d040f1984352b5c38fdd850a85ba8
Sep 28 02:55:34 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:34.518051 10316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:34.498059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:42771 (local address 127.27.229.254:46197)
0928 02:55:34.498582 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:34.498588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:34.499772 (+ 1184us) server_negotiation.cc:407] Connection header received
0928 02:55:34.501766 (+ 1994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:34.501775 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:34.501868 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:34.501980 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:34.503575 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:34.504799 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:34.506937 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:34.507383 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:34.507500 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:34.512079 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:34.512084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:34.512088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:34.512123 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:34.514803 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:34.515663 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:34.515667 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:34.515669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:34.515820 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:34.516279 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:34.516288 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:34.516290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:34.516552 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:34.516719 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:34.517434 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 02:55:34.517678 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":260,"threads_started":1}
I20260928 02:55:34.519088 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:34.498689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:42771)
0928 02:55:34.499595 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:34.499646 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:34.501463 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:34.502182 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:34.502190 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:34.502748 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:34.503392 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:34.503410 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:34.504990 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:34.504995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:34.506669 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:34.506688 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:34.507642 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:34.507651 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:34.508018 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:34.508795 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:34.508830 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:34.511812 (+ 2982us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:34.515034 (+ 3222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:34.515044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:34.515064 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:34.515499 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:34.515944 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:34.515949 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:34.515953 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:34.516157 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:34.516688 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:34.516696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:34.517065 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:55:34.517344 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:55:34.517894 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":388,"threads_started":1}
I20260928 02:55:34.520743 10314 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:55:34.521036 10314 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:34.521740 10314 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
I20260928 02:55:34.523458 9991 ts_manager.cc:194] Registered new tserver with Master: 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:55:34.524550 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:42771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:55:34.554077 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:34.888644 10320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:34.888932 10320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:34.889106 10320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:34.896725 10320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:34.896850 10320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:34.897056 10320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:34.908035 10320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:34.909478 10320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:34.910977 10320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:34.925101 10326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:34.925058 10328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:34.925154 10325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:34.925921 10320 server_base.cc:1034] running on GCE node
I20260928 02:55:34.926592 10320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:34.927743 10320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:34.929013 10320 hybrid_clock.cc:648] HybridClock initialized: now 1790564134928947 us; error 63 us; skew 500 ppm
I20260928 02:55:34.929409 10320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:34 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:34.932915 10320 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:34.934640 10320 webserver.cc:460] Webserver started at http://127.27.229.195:35239/ using document root <none> and password file <none>
I20260928 02:55:34.935470 10320 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:34.935627 10320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:34.935951 10320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:34.939208 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "ad389a62e2d14f5899a1fbc97b795536"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "8628207532c6c2b21b40cb4fc4925752"
server_key_iv: "fa3528ace73581e09dfd74c9cec1de93"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.940062 10320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "ad389a62e2d14f5899a1fbc97b795536"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "8628207532c6c2b21b40cb4fc4925752"
server_key_iv: "fa3528ace73581e09dfd74c9cec1de93"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.945770 10320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:55:34.949538 10335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.951216 10320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:34.951411 10320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "ad389a62e2d14f5899a1fbc97b795536"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "8628207532c6c2b21b40cb4fc4925752"
server_key_iv: "fa3528ace73581e09dfd74c9cec1de93"
server_key_version: "encryptionkey@0"
I20260928 02:55:34.951634 10320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:34.977558 10320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:34.982286 10320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:34.982596 10320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:34.983704 10320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:34.985127 10320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:34.985229 10320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.985383 10320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:34.985455 10320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:34.993736 10180 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:35.021508 10320 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:41015
I20260928 02:55:35.021539 10447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:41015 every 8 connection(s)
I20260928 02:55:35.022809 10320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:35.029625 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10320
I20260928 02:55:35.029860 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:55:35.030355 28567 external_mini_cluster.cc:1428] Setting key ac020a5f18ece898316ae165eeb87d78
Sep 28 02:55:35 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564134, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:35.044137 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.025583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:36857 (local address 127.27.229.254:46197)
0928 02:55:35.026062 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.026069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.027012 (+ 943us) server_negotiation.cc:407] Connection header received
0928 02:55:35.028866 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.028871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.028961 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.029076 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:35.031153 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.031987 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.033605 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.034021 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.034160 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.037892 (+ 3732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:35.037899 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:35.037902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:35.037944 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:35.040770 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.041631 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.041635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.041638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.041766 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.042216 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.042221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.042223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.042498 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:35.042668 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.043451 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.043697 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":224,"threads_started":1}
I20260928 02:55:35.045281 10451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.026060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:36857)
0928 02:55:35.026801 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.026903 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.028569 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.029537 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.029545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.030230 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:35.030961 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.030982 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.032170 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.032176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.033448 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.033460 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.034264 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.034271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.034513 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.035272 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:35.035294 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:35.037651 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:35.041014 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:35.041023 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:35.041038 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:35.041482 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:35.041884 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:35.041888 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:35.041892 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:35.042088 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:35.042642 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:35.042650 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:35.043081 (+ 431us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.043353 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.043901 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":272,"threads_started":1}
I20260928 02:55:35.046651 10448 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
I20260928 02:55:35.047011 10448 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:35.047739 10448 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
I20260928 02:55:35.049144 9991 ts_manager.cc:194] Registered new tserver with Master: ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
I20260928 02:55:35.049974 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:36857
I20260928 02:55:35.058643 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:55:35.078052 10450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.063396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52620 (local address 127.27.229.254:46197)
0928 02:55:35.063581 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.063587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.063977 (+ 390us) server_negotiation.cc:407] Connection header received
0928 02:55:35.064125 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.064129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.064196 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.064285 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:35.066593 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.067745 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.069198 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.069625 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.069802 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.072338 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:35.072343 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:35.072346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:35.072384 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:35.075244 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.076014 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.076019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.076021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.076203 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.076556 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.076559 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.076561 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.076904 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:35.077085 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.077596 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.077818 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 02:55:35.082367 9991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:55:35.089798 9991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:55:35.116099 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.103815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56910 (local address 127.27.229.195:41015)
0928 02:55:35.104823 (+ 1008us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.104837 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.106968 (+ 2131us) server_negotiation.cc:407] Connection header received
0928 02:55:35.107139 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.107147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.107376 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.107582 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:35.108984 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.110421 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.112903 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.114340 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.114520 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.114705 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.115558 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.115739 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":523,"threads_started":1}
I20260928 02:55:35.116906 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.103695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:41015 (local address 127.0.0.1:56910)
0928 02:55:35.106832 (+ 3137us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.106868 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.106993 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.108322 (+ 1329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.108332 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.108358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:35.108784 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.108796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.110627 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.110631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.112685 (+ 2054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.112700 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.115008 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.115016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.115270 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.115324 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.116330 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.116466 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3003,"mutex_wait_us":48,"thread_start_us":2834,"threads_started":1}
I20260928 02:55:35.117012 10461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.103579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41937 (local address 127.0.0.1:41198)
0928 02:55:35.106287 (+ 2708us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.106350 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.106679 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.107429 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.107436 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.107467 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:35.107872 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.107887 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.109945 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.109953 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.113278 (+ 3325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.113300 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.115498 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.115504 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.115664 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.115697 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.115777 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.115988 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2364,"mutex_wait_us":27,"thread_start_us":2184,"threads_started":1}
I20260928 02:55:35.117969 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.104334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41198 (local address 127.27.229.194:41937)
0928 02:55:35.105955 (+ 1621us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.105963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.106501 (+ 538us) server_negotiation.cc:407] Connection header received
0928 02:55:35.106963 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.106974 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.107218 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.107428 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:35.108057 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.109791 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.113480 (+ 3689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.115030 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.115193 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.115342 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.117463 (+ 2121us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.117654 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1294,"threads_started":1}
I20260928 02:55:35.121866 10249 tablet_service.cc:1467] Processing CreateTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 (DEFAULT_TABLE table=test-table [id=a272fa7e3bea4e25be42969c23a36925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:35.121829 10383 tablet_service.cc:1467] Processing CreateTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 (DEFAULT_TABLE table=test-table [id=a272fa7e3bea4e25be42969c23a36925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:35.123488 10249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620c8a68aa42ca8133143a2d416fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:35.123519 10383 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620c8a68aa42ca8133143a2d416fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:35.125869 10466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.107683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:59090)
0928 02:55:35.115446 (+ 7763us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.115477 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.115636 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.118505 (+ 2869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.118508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.118529 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:35.118788 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.118795 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.120385 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.120391 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.122918 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.122937 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.125016 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.125027 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.125205 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.125245 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.125343 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.125496 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7570,"thread_start_us":7410,"threads_started":1}
I20260928 02:55:35.125967 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.107320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59090 (local address 127.27.229.193:39057)
0928 02:55:35.117593 (+ 10273us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.117602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.117631 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:55:35.117733 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.117744 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.118052 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.118280 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:35.119018 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.120227 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.123138 (+ 2911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.124691 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.124919 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.125098 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.125448 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.125635 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10084,"thread_start_us":9972,"threads_started":1}
I20260928 02:55:35.130954 10115 tablet_service.cc:1467] Processing CreateTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 (DEFAULT_TABLE table=test-table [id=a272fa7e3bea4e25be42969c23a36925]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:35.132959 10115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4620c8a68aa42ca8133143a2d416fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:35.135829 10467 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Bootstrap starting.
I20260928 02:55:35.137910 10468 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Bootstrap starting.
I20260928 02:55:35.140430 10467 tablet_bootstrap.cc:654] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:35.141517 10468 tablet_bootstrap.cc:654] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:35.141779 10467 log.cc:824] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:35.142737 10468 log.cc:824] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:35.145022 10467 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: No bootstrap required, opened a new log
I20260928 02:55:35.145334 10467 ts_tablet_manager.cc:1397] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 02:55:35.145766 10468 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: No bootstrap required, opened a new log
I20260928 02:55:35.146029 10468 ts_tablet_manager.cc:1397] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 02:55:35.147954 10471 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Bootstrap starting.
I20260928 02:55:35.151081 10471 tablet_bootstrap.cc:654] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:35.152108 10471 log.cc:824] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:35.154816 10471 tablet_bootstrap.cc:492] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: No bootstrap required, opened a new log
I20260928 02:55:35.155217 10471 ts_tablet_manager.cc:1397] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260928 02:55:35.156715 10467 raft_consensus.cc:348] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.156715 10468 raft_consensus.cc:348] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.157171 10468 raft_consensus.cc:374] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:35.157168 10467 raft_consensus.cc:374] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:35.157279 10467 raft_consensus.cc:729] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37468f93c61143a49e27aadf1258e984, State: Initialized, Role: FOLLOWER
I20260928 02:55:35.157313 10468 raft_consensus.cc:729] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad389a62e2d14f5899a1fbc97b795536, State: Initialized, Role: FOLLOWER
I20260928 02:55:35.157922 10468 consensus_queue.cc:260] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.157922 10467 consensus_queue.cc:260] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.164018 10471 raft_consensus.cc:348] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.164412 10471 raft_consensus.cc:374] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:35.164577 10471 raft_consensus.cc:729] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406a33188efa422e8ec38541b4a1b812, State: Initialized, Role: FOLLOWER
I20260928 02:55:35.165172 10314 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:35.165666 10448 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:35.166116 10467 ts_tablet_manager.cc:1428] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Time spent starting tablet: real 0.021s user 0.012s sys 0.009s
I20260928 02:55:35.166160 10468 ts_tablet_manager.cc:1428] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Time spent starting tablet: real 0.020s user 0.008s sys 0.011s
I20260928 02:55:35.166689 10471 consensus_queue.cc:260] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.168510 10471 ts_tablet_manager.cc:1428] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Time spent starting tablet: real 0.013s user 0.007s sys 0.003s
W20260928 02:55:35.212838 10181 tablet.cc:2369] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:55:35.247463 10315 tablet.cc:2369] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:55:35.274701 10449 tablet.cc:2369] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:35.345299 10475 raft_consensus.cc:482] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:35.345667 10475 raft_consensus.cc:504] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.347808 10475 leader_election.cc:288] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015), 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937)
I20260928 02:55:35.355997 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.348590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:56989 (local address 127.27.229.194:41937)
0928 02:55:35.348775 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.348782 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.349225 (+ 443us) server_negotiation.cc:407] Connection header received
0928 02:55:35.349555 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.349560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.349642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.349712 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:35.350558 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.351650 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.353969 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.354771 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.355011 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.355089 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.355565 (+ 476us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.355707 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:55:35.356063 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.348212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41937 (local address 127.27.229.193:56989)
0928 02:55:35.349093 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.349121 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.349347 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.349936 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.349939 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.349975 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:35.350327 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.350335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.351789 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.351794 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.353826 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.353840 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.355047 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.355053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.355210 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.355266 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.355506 (+ 240us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.355667 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":504,"threads_started":1}
I20260928 02:55:35.357091 10269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620c8a68aa42ca8133143a2d416fb7" candidate_uuid: "406a33188efa422e8ec38541b4a1b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37468f93c61143a49e27aadf1258e984" is_pre_election: true
I20260928 02:55:35.357452 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.348364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:58637 (local address 127.27.229.195:41015)
0928 02:55:35.348581 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.348590 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.349424 (+ 834us) server_negotiation.cc:407] Connection header received
0928 02:55:35.349571 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.349576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.349648 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.349754 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:35.350456 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.351322 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.353906 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.355168 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.355327 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.355412 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.357063 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.357199 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:55:35.357558 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.348222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:41015 (local address 127.27.229.193:58637)
0928 02:55:35.349297 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:35.349314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:35.349439 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:35.349792 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:35.349797 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:35.349821 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:35.350270 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.350281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.351460 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.351467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.353721 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:35.353740 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.355436 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.355442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:35.356885 (+ 1443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.356924 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 02:55:35.357032 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:55:35.357176 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"mutex_wait_us":192,"thread_start_us":527,"threads_started":1}
I20260928 02:55:35.357668 10269 raft_consensus.cc:2454] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406a33188efa422e8ec38541b4a1b812 in term 0.
I20260928 02:55:35.358443 10068 leader_election.cc:302] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37468f93c61143a49e27aadf1258e984, 406a33188efa422e8ec38541b4a1b812; no voters:
I20260928 02:55:35.358505 10403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620c8a68aa42ca8133143a2d416fb7" candidate_uuid: "406a33188efa422e8ec38541b4a1b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad389a62e2d14f5899a1fbc97b795536" is_pre_election: true
I20260928 02:55:35.358944 10475 raft_consensus.cc:2781] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:55:35.359141 10475 raft_consensus.cc:482] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:55:35.359117 10403 raft_consensus.cc:2454] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 406a33188efa422e8ec38541b4a1b812 in term 0.
I20260928 02:55:35.359310 10475 raft_consensus.cc:3037] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:35.362576 10475 raft_consensus.cc:504] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.363881 10403 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620c8a68aa42ca8133143a2d416fb7" candidate_uuid: "406a33188efa422e8ec38541b4a1b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad389a62e2d14f5899a1fbc97b795536"
I20260928 02:55:35.364060 10269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4620c8a68aa42ca8133143a2d416fb7" candidate_uuid: "406a33188efa422e8ec38541b4a1b812" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37468f93c61143a49e27aadf1258e984"
I20260928 02:55:35.364266 10269 raft_consensus.cc:3037] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:35.364310 10403 raft_consensus.cc:3037] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:35.367909 10269 raft_consensus.cc:2454] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406a33188efa422e8ec38541b4a1b812 in term 1.
I20260928 02:55:35.368479 10068 leader_election.cc:302] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37468f93c61143a49e27aadf1258e984, 406a33188efa422e8ec38541b4a1b812; no voters:
I20260928 02:55:35.369954 10475 leader_election.cc:288] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 1 election: Requested vote from peers ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015), 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937)
I20260928 02:55:35.370169 10478 raft_consensus.cc:2781] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:35.370393 10478 raft_consensus.cc:686] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 1 LEADER]: Becoming Leader. State: Replica: 406a33188efa422e8ec38541b4a1b812, State: Running, Role: LEADER
I20260928 02:55:35.371038 10478 consensus_queue.cc:237] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:35.376422 10403 raft_consensus.cc:2454] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 406a33188efa422e8ec38541b4a1b812 in term 1.
I20260928 02:55:35.376240 9990 catalog_manager.cc:5495] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 reported cstate change: term changed from 0 to 1, leader changed from <none> to 406a33188efa422e8ec38541b4a1b812 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "406a33188efa422e8ec38541b4a1b812" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } health_report { overall_health: HEALTHY } } }
I20260928 02:55:35.417626 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.412435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59102 (local address 127.27.229.193:39057)
0928 02:55:35.412636 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.412641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.412660 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:55:35.412752 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.412755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.412830 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.412963 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:35.413339 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.414057 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.415500 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.415897 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.415993 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.416482 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:35.416999 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:35.417108 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.417252 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.417320 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:55:35.426282 10403 raft_consensus.cc:1261] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Refusing update from remote peer 406a33188efa422e8ec38541b4a1b812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:35.426281 10269 raft_consensus.cc:1261] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Refusing update from remote peer 406a33188efa422e8ec38541b4a1b812: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:35.427423 10475 consensus_queue.cc:1035] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:35.427760 10478 consensus_queue.cc:1035] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:35.439502 10482 mvcc.cc:205] Tried to move back new op lower bound from 7334150698694848512 to 7334150698484944896. Current Snapshot: MvccSnapshot[applied={T|T < 7334150698694848512}]
I20260928 02:55:35.441257 10483 mvcc.cc:205] Tried to move back new op lower bound from 7334150698694848512 to 7334150698484944896. Current Snapshot: MvccSnapshot[applied={T|T < 7334150698694848512}]
I20260928 02:55:35.453068 9991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:35.453421 9991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52620:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:35.457653 9991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Sending DeleteTablet for 3 replicas of tablet d4620c8a68aa42ca8133143a2d416fb7
I20260928 02:55:35.459052 10249 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:35 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:41198
I20260928 02:55:35.459053 10115 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:35 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:59090
I20260928 02:55:35.459199 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10052
I20260928 02:55:35.460088 10383 tablet_service.cc:1514] Processing DeleteTablet for tablet d4620c8a68aa42ca8133143a2d416fb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:35 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56910
I20260928 02:55:35.471962 10490 tablet_replica.cc:330] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: stopping tablet replica
I20260928 02:55:35.472674 10490 raft_consensus.cc:2229] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:35.473052 10491 tablet_replica.cc:330] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: stopping tablet replica
I20260928 02:55:35.473361 10490 raft_consensus.cc:2258] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:35.474515 10491 raft_consensus.cc:2229] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:55:35.475407 10493 tablet_replica.cc:330] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: stopping tablet replica
I20260928 02:55:35.475857 10490 ts_tablet_manager.cc:1905] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:35.475836 10493 raft_consensus.cc:2229] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:35.476176 10491 raft_consensus.cc:2258] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:35.476344 10493 raft_consensus.cc:2258] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:35.479957 10491 ts_tablet_manager.cc:1905] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:35.481972 10493 ts_tablet_manager.cc:1905] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:35.484043 10490 ts_tablet_manager.cc:1918] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:35.484305 10490 log.cc:1196] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/d4620c8a68aa42ca8133143a2d416fb7
I20260928 02:55:35.484929 10490 ts_tablet_manager.cc:1939] T d4620c8a68aa42ca8133143a2d416fb7 P 37468f93c61143a49e27aadf1258e984: Deleting consensus metadata
I20260928 02:55:35.487049 9975 catalog_manager.cc:4841] TS 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): tablet d4620c8a68aa42ca8133143a2d416fb7 (table test-table [id=a272fa7e3bea4e25be42969c23a36925]) successfully deleted
Sep 28 02:55:35 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:55:35.493814 10491 ts_tablet_manager.cc:1918] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:35.494132 10491 log.cc:1196] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/d4620c8a68aa42ca8133143a2d416fb7
I20260928 02:55:35.494205 10493 ts_tablet_manager.cc:1918] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:35.494417 10493 log.cc:1196] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/d4620c8a68aa42ca8133143a2d416fb7
I20260928 02:55:35.494812 10491 ts_tablet_manager.cc:1939] T d4620c8a68aa42ca8133143a2d416fb7 P 406a33188efa422e8ec38541b4a1b812: Deleting consensus metadata
I20260928 02:55:35.495025 10493 ts_tablet_manager.cc:1939] T d4620c8a68aa42ca8133143a2d416fb7 P ad389a62e2d14f5899a1fbc97b795536: Deleting consensus metadata
I20260928 02:55:35.496850 9977 catalog_manager.cc:4841] TS 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057): tablet d4620c8a68aa42ca8133143a2d416fb7 (table test-table [id=a272fa7e3bea4e25be42969c23a36925]) successfully deleted
I20260928 02:55:35.497308 9977 catalog_manager.cc:4841] TS ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): tablet d4620c8a68aa42ca8133143a2d416fb7 (table test-table [id=a272fa7e3bea4e25be42969c23a36925]) successfully deleted
I20260928 02:55:35.502727 10465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.460465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.27.229.193:39057)
0928 02:55:35.460653 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.460669 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.477579 (+ 16910us) server_negotiation.cc:407] Connection header received
0928 02:55:35.478223 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.478231 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.478325 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.478415 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:35.480638 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.481884 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.484058 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.484833 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.484937 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.490538 (+ 5601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:35.490554 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:35.490601 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:35.490652 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:35.499123 (+ 8471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.500129 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.500139 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.500143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.500327 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.500797 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.500802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.500807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.501353 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:35.501743 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.502048 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.502317 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:55:35.503538 10175 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:35.609669 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10052
I20260928 02:55:35.628573 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10186
Sep 28 02:55:35 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:55:35.645543 10463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.629398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41204 (local address 127.27.229.194:41937)
0928 02:55:35.629565 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.629570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.629589 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:55:35.629778 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.629782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.629861 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.629953 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:35.631321 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.632076 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.633380 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.633772 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.633870 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.638904 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:35.638936 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:35.638953 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:35.638997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:35.641957 (+ 2960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.643265 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.643273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.643276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.643434 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.643906 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.643910 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.643912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.644415 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:35.644645 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.645036 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.645260 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:55:35.646337 10309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:35.737402 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10186
I20260928 02:55:35.754614 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10320
Sep 28 02:55:35 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:55:35.770710 10462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:35.755464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56916 (local address 127.27.229.195:41015)
0928 02:55:35.755669 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:35.755674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:35.755693 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:55:35.755899 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:35.755904 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:35.755970 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:35.756056 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:35.757379 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.758081 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.759585 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:35.759988 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:35.760093 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:35.764595 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:35.764611 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:35.764627 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:35.764669 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:35.767984 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.768658 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.768665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.768668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.768823 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:35.769174 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:35.769177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:35.769180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:35.769554 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:35.769781 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:35.770161 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 02:55:35.770326 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:55:35.771457 10443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:35.862354 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10320
I20260928 02:55:35.879726 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 9959
I20260928 02:55:35.881040 10021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:35.993637 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 9959
I20260928 02:55:36.015280 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--webserver_interface=127.27.229.254
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:36.356192 10498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:36.356500 10498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:36.356667 10498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:36.364699 10498 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:36.364814 10498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:36.364898 10498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:36.364976 10498 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:36.365037 10498 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:36.375278 10498 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:36.376646 10498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:36.377986 10498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:36.402328 10507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:36.402458 10504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:36.402442 10505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:36.404095 10498 server_base.cc:1034] running on GCE node
I20260928 02:55:36.405083 10498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:36.406883 10498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:36.408197 10498 hybrid_clock.cc:648] HybridClock initialized: now 1790564136408091 us; error 66 us; skew 500 ppm
I20260928 02:55:36.408634 10498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:36 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564136, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:36.412667 10498 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:36.414479 10498 webserver.cc:460] Webserver started at http://127.27.229.254:37513/ using document root <none> and password file <none>
I20260928 02:55:36.415361 10498 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:36.415531 10498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:36.422073 10498 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:55:36.425431 10514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:36.426815 10498 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:55:36.427071 10498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "cb3cc013b5634f36bd62bc3561a04696"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "c33cbd0e678ef94bb33854303f397a7f"
server_key_iv: "a2de3bfad0e018ef3c05a93298871eda"
server_key_version: "encryptionkey@0"
I20260928 02:55:36.428231 10498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:36.444945 10498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:36.449959 10498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:36.450299 10498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:36.471340 10498 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:46197
I20260928 02:55:36.471347 10565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:46197 every 8 connection(s)
I20260928 02:55:36.472967 10498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:36.479416 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10498
I20260928 02:55:36.480338 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:39057
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=36485
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:36.480696 10566 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:55:36.484390 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap starting.
I20260928 02:55:36.513854 10566 log.cc:824] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:36.531039 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:55:36.531752 10566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap complete.
I20260928 02:55:36.545415 10566 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:36.546373 10566 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Initialized, Role: FOLLOWER
I20260928 02:55:36.547207 10566 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:36.547477 10566 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:36.547626 10566 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:36.547804 10566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:55:36.552734 10566 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:36.553373 10566 leader_election.cc:302] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3cc013b5634f36bd62bc3561a04696; no voters:
I20260928 02:55:36.554042 10566 leader_election.cc:288] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:55:36.554183 10570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:55:36.555542 10570 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 LEADER]: Becoming Leader. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Running, Role: LEADER
I20260928 02:55:36.556196 10570 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:36.556617 10566 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:36.558797 10572 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3cc013b5634f36bd62bc3561a04696. Latest consensus state: current_term: 2 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:36.559201 10572 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:36.559659 10571 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:36.560003 10571 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:36.560300 10576 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:36.583122 10576 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:36.585880 10576 catalog_manager.cc:1259] Loaded cluster ID: 12715fb200b24911865280ca0b18e6b9
I20260928 02:55:36.586123 10576 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:36.590477 10576 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 02:55:36.609537 10587 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:55:36.613482 10576 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 02:55:36.620999 10576 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:36.622684 10576 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Loaded TSK: 0
I20260928 02:55:36.623889 10576 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:55:36.891201 10568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:36.891566 10568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:36.891731 10568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:36.899767 10568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:36.899927 10568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:36.900120 10568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:36.910477 10568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:39057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=36485
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:36.911922 10568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:36.913303 10568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:36.929090 10593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:36.929759 10594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:36.929955 10596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:36.931318 10568 server_base.cc:1034] running on GCE node
I20260928 02:55:36.932003 10568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:36.933089 10568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:36.934383 10568 hybrid_clock.cc:648] HybridClock initialized: now 1790564136934317 us; error 52 us; skew 500 ppm
I20260928 02:55:36.934768 10568 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:36 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564136, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:36.942536 10568 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:36.944383 10568 webserver.cc:460] Webserver started at http://127.27.229.193:36485/ using document root <none> and password file <none>
I20260928 02:55:36.945129 10568 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:36.945295 10568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:36.951165 10568 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:55:36.954021 10603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:36.955587 10568 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:36.955785 10568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "406a33188efa422e8ec38541b4a1b812"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "d5e3b2ea865bd51be97ac31714773631"
server_key_iv: "513110edede80db64a8c132125d215e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:36.956848 10568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:36.978761 10568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:36.983469 10568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:36.983774 10568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:36.984830 10568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:36.986194 10568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:36.986281 10568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:36.986423 10568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:36.986506 10568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.021872 10568 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39057
I20260928 02:55:37.021921 10715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39057 every 8 connection(s)
I20260928 02:55:37.023555 10568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:37.030220 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10568
I20260928 02:55:37.031061 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:41937
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=46873
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:37 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564136, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:37.050654 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.026502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:34617 (local address 127.27.229.254:46197)
0928 02:55:37.027099 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:37.027110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:37.027852 (+ 742us) server_negotiation.cc:407] Connection header received
0928 02:55:37.030219 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:37.030237 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:37.030724 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:37.031198 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:37.032924 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.035107 (+ 2183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.036750 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.037874 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.038115 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.042668 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:37.042678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:37.042694 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:37.042737 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:37.045865 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.046710 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.046718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.046723 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.046959 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.047414 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.047420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.047423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.047921 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:37.048136 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:37.048915 (+ 779us) server_negotiation.cc:299] Negotiation successful
0928 02:55:37.049304 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":243,"threads_started":1}
I20260928 02:55:37.050654 10719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.026813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:34617)
0928 02:55:37.027675 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:37.027721 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:37.029458 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:37.031383 (+ 1925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:37.031393 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:37.031959 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:37.032669 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.032696 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.035301 (+ 2605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.035306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.036578 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.036591 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.038167 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.038177 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.038513 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.039431 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:37.039466 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:37.042336 (+ 2870us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:37.046058 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.046068 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.046085 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.046542 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.047091 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.047096 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.047100 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.047296 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.048082 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:37.048091 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:37.048431 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 02:55:37.048714 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 02:55:37.049303 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":343,"threads_started":1}
I20260928 02:55:37.054018 10716 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
I20260928 02:55:37.054361 10716 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:37.055198 10716 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
I20260928 02:55:37.057621 10531 ts_manager.cc:194] Registered new tserver with Master: 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057)
I20260928 02:55:37.059228 10531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:34617
W20260928 02:55:37.374644 10720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:37.375051 10720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:37.375212 10720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:37.382893 10720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:37.383056 10720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:37.383250 10720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:37.393330 10720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=46873
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:37.394706 10720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:37.396232 10720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:37.410514 10728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:37.410457 10725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:37.410554 10726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:37.411530 10720 server_base.cc:1034] running on GCE node
I20260928 02:55:37.412303 10720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:37.413383 10720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:37.414669 10720 hybrid_clock.cc:648] HybridClock initialized: now 1790564137414625 us; error 63 us; skew 500 ppm
I20260928 02:55:37.415076 10720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:37 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564137, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:37.418558 10720 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:37.420255 10720 webserver.cc:460] Webserver started at http://127.27.229.194:46873/ using document root <none> and password file <none>
I20260928 02:55:37.421025 10720 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:37.421208 10720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:37.427101 10720 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:55:37.429919 10735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.431447 10720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:37.431701 10720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "37468f93c61143a49e27aadf1258e984"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "85372e2533ae1f017612d7f27a827182"
server_key_iv: "c4859c89fc46d6ecc1d6319ebb717bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:37.432842 10720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:37.445875 10720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:37.450407 10720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:37.450672 10720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:37.451854 10720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:37.453271 10720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:37.453362 10720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.453502 10720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:37.453590 10720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.496150 10720 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41937
I20260928 02:55:37.496248 10847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41937 every 8 connection(s)
I20260928 02:55:37.497817 10720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:37.507351 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10720
I20260928 02:55:37.508104 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:41015
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=35239
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:55:37 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564137, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:37.520128 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.500598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:39787 (local address 127.27.229.254:46197)
0928 02:55:37.500828 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:37.500832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:37.501890 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 02:55:37.503853 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:37.503858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:37.503956 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:37.504066 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:37.505654 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.506368 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.508153 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.508930 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.509094 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.513250 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:37.513270 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:37.513277 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:37.513330 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:37.516478 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.517544 (+ 1066us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.517548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.517550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.517677 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.518201 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.518205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.518208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.518531 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:37.518672 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:37.519684 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0928 02:55:37.519883 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260928 02:55:37.521616 10850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.500953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:39787)
0928 02:55:37.501712 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:37.501752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:37.503542 (+ 1790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:37.504349 (+ 807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:37.504357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:37.504878 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:37.505454 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.505472 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.506568 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.506572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.507987 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.508001 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.509201 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.509208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.509442 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.510259 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:37.510285 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:37.512976 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:37.516657 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.516672 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.516693 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.517383 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.517790 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.517797 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.517803 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.518049 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.518683 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:37.518694 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:37.519203 (+ 509us) client_negotiation.cc:769] Sending connection context
0928 02:55:37.519533 (+ 330us) client_negotiation.cc:240] Negotiation successful
0928 02:55:37.520124 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":326,"threads_started":1}
I20260928 02:55:37.523315 10848 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
I20260928 02:55:37.523689 10848 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:37.524446 10848 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
I20260928 02:55:37.526098 10531 ts_manager.cc:194] Registered new tserver with Master: 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937)
I20260928 02:55:37.527489 10531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:39787
W20260928 02:55:37.843086 10851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:37.843393 10851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:37.843492 10851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:37.850961 10851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:37.851089 10851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:37.851205 10851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:37.862309 10851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:41015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=35239
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:37.863780 10851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:37.865206 10851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:37.879743 10859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:37.879874 10856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:37.879850 10857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:37.880797 10851 server_base.cc:1034] running on GCE node
I20260928 02:55:37.881479 10851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:37.882562 10851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:37.883826 10851 hybrid_clock.cc:648] HybridClock initialized: now 1790564137883757 us; error 86 us; skew 500 ppm
I20260928 02:55:37.884193 10851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:37 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564137, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:37.887632 10851 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:37.889411 10851 webserver.cc:460] Webserver started at http://127.27.229.195:35239/ using document root <none> and password file <none>
I20260928 02:55:37.890187 10851 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:37.890362 10851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:37.896268 10851 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:55:37.899480 10866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.901059 10851 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260928 02:55:37.901280 10851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "ad389a62e2d14f5899a1fbc97b795536"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "8628207532c6c2b21b40cb4fc4925752"
server_key_iv: "fa3528ace73581e09dfd74c9cec1de93"
server_key_version: "encryptionkey@0"
I20260928 02:55:37.902441 10851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:37.926357 10851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:37.931277 10851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:37.931584 10851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:37.932787 10851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:37.934301 10851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:37.934425 10851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.934563 10851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:37.934662 10851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:37.969317 10851 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:41015
I20260928 02:55:37.969374 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:41015 every 8 connection(s)
I20260928 02:55:37.970887 10851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:37.975351 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 10851
Sep 28 02:55:37 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564137, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:37.992879 10983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.976404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52628 (local address 127.27.229.254:46197)
0928 02:55:37.977158 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:37.977164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:37.977195 (+ 31us) server_negotiation.cc:407] Connection header received
0928 02:55:37.977282 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:37.977287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:37.977366 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:37.977483 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:37.979193 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.980374 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.982232 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.982963 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.983099 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.986175 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:37.986183 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:37.986188 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:37.986232 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:37.989493 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.990489 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.990493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.990495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.990645 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.991129 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.991132 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.991134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.991484 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:37.991693 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:37.992209 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 02:55:37.992458 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":532,"threads_started":1}
I20260928 02:55:37.995967 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.973584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:38801 (local address 127.27.229.254:46197)
0928 02:55:37.973791 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:37.973796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:37.975125 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 02:55:37.977935 (+ 2810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:37.977940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:37.978010 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:37.978121 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:37.980191 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.981268 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.983494 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.984185 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.984317 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.988427 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:37.988433 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:37.988437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:37.988483 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:37.991890 (+ 3407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.992959 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.992965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.992968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.993161 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:37.993700 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:37.993706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:37.993710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:37.994069 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:37.994216 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:37.995343 (+ 1127us) server_negotiation.cc:299] Negotiation successful
0928 02:55:37.995568 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:55:37.996932 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:37.974046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:38801)
0928 02:55:37.974927 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:37.974990 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:37.977640 (+ 2650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:37.978376 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:37.978385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:37.979009 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:37.979956 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.979983 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.981492 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.981496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.983344 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:37.983357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:37.984421 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:37.984427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:37.984652 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:37.985354 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:37.985383 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:37.988164 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:37.992085 (+ 3921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.992096 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.992117 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.992796 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.993287 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:37.993292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:37.993297 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:37.993569 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:37.994219 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:37.994228 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:37.994919 (+ 691us) client_negotiation.cc:769] Sending connection context
0928 02:55:37.995199 (+ 280us) client_negotiation.cc:240] Negotiation successful
0928 02:55:37.995646 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":321,"threads_started":1}
I20260928 02:55:37.998512 10979 heartbeater.cc:347] Connected to a master server at 127.27.229.254:46197
I20260928 02:55:37.998915 10979 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:37.999878 10979 heartbeater.cc:511] Master 127.27.229.254:46197 requested a full tablet report, sending...
I20260928 02:55:38.001483 10531 ts_manager.cc:194] Registered new tserver with Master: ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
I20260928 02:55:38.002358 10531 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:38801
I20260928 02:55:38.008880 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:55:38.016129 10718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.009960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52640 (local address 127.27.229.254:46197)
0928 02:55:38.010256 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.010262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.010570 (+ 308us) server_negotiation.cc:407] Connection header received
0928 02:55:38.010881 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.010897 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.011040 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.011234 (+ 194us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:38.011674 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.012435 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.013899 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.014332 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.014474 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.015033 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:38.015542 (+ 509us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:38.015714 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.015766 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.015856 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 02:55:38.017313 10531 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:52640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:55:38.025130 10531 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:55:38.062680 10716 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:38.066200 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.039279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56926 (local address 127.27.229.195:41015)
0928 02:55:38.055083 (+ 15804us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.055089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.055115 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:55:38.055236 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.055243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.055447 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.055626 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:38.056626 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.058309 (+ 1683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.061975 (+ 3666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.063971 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.064201 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.064830 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.065643 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.065834 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15628,"thread_start_us":15448,"threads_started":1}
I20260928 02:55:38.066429 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.038770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41210 (local address 127.27.229.194:41937)
0928 02:55:38.053864 (+ 15094us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.053874 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.054585 (+ 711us) server_negotiation.cc:407] Connection header received
0928 02:55:38.054693 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.054701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.054996 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.055216 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:38.056447 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.058309 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.063113 (+ 4804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.064602 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.064793 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.064960 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.065850 (+ 890us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.066075 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14888,"thread_start_us":15117,"threads_started":1}
I20260928 02:55:38.066461 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.038769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:41015 (local address 127.0.0.1:56926)
0928 02:55:38.053997 (+ 15228us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.054051 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:38.054312 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:38.055927 (+ 1615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:38.055931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:38.055958 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:38.056425 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.056436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.058479 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.058485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.061809 (+ 3324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.061825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.064976 (+ 3151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.064982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.065249 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.065429 (+ 180us) client_negotiation.cc:769] Sending connection context
0928 02:55:38.065586 (+ 157us) client_negotiation.cc:240] Negotiation successful
0928 02:55:38.065921 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15006,"mutex_wait_us":536,"spinlock_wait_cycles":22272,"thread_start_us":15875,"threads_started":1}
I20260928 02:55:38.067196 10990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.038767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41937 (local address 127.0.0.1:41210)
0928 02:55:38.053994 (+ 15227us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.054050 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:38.054312 (+ 262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:38.055639 (+ 1327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:38.055646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:38.055674 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:38.056167 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.056195 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.058479 (+ 2284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.058485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.062910 (+ 4425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.062926 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.065122 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.065127 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.065429 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.065474 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 02:55:38.065579 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:55:38.066706 (+ 1127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14921,"mutex_wait_us":2248,"spinlock_wait_cycles":12032,"thread_start_us":15319,"threads_started":1}
I20260928 02:55:38.070444 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.055677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:59118)
0928 02:55:38.061015 (+ 5338us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.061035 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:38.061225 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:38.062143 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:38.062148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:38.062175 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:38.062671 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.062684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.065281 (+ 2597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.065287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.067581 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.067597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.069592 (+ 1995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.069599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.069780 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.069813 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:55:38.069916 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:55:38.070049 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5188,"mutex_wait_us":49,"thread_start_us":5032,"threads_started":1}
I20260928 02:55:38.070461 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.055823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59118 (local address 127.27.229.193:39057)
0928 02:55:38.061487 (+ 5664us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.061496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.061525 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:55:38.061621 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.061628 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.061924 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.062110 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:38.062938 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.065148 (+ 2210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.067789 (+ 2641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.069340 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.069547 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.069725 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.069979 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.070142 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5457,"thread_start_us":4414,"threads_started":1}
I20260928 02:55:38.070783 10783 tablet_service.cc:1467] Processing CreateTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac (DEFAULT_TABLE table=test-table [id=428172a78fcb43ca82833c0c35d6b192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:38.070782 10914 tablet_service.cc:1467] Processing CreateTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac (DEFAULT_TABLE table=test-table [id=428172a78fcb43ca82833c0c35d6b192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:38.072242 10783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ce2c1fc90343828734d7fbb91a57ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:38.072274 10914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ce2c1fc90343828734d7fbb91a57ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:38.075167 10651 tablet_service.cc:1467] Processing CreateTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac (DEFAULT_TABLE table=test-table [id=428172a78fcb43ca82833c0c35d6b192]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:38.076715 10651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50ce2c1fc90343828734d7fbb91a57ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:38.086807 10995 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Bootstrap starting.
I20260928 02:55:38.091333 10995 tablet_bootstrap.cc:654] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:38.091627 10996 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Bootstrap starting.
I20260928 02:55:38.092808 10995 log.cc:824] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:38.095868 10998 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Bootstrap starting.
I20260928 02:55:38.096207 10996 tablet_bootstrap.cc:654] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:38.096446 10995 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: No bootstrap required, opened a new log
I20260928 02:55:38.096758 10995 ts_tablet_manager.cc:1397] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260928 02:55:38.097604 10996 log.cc:824] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:38.099448 10998 tablet_bootstrap.cc:654] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:38.100509 10998 log.cc:824] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:38.100944 10996 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: No bootstrap required, opened a new log
I20260928 02:55:38.101238 10996 ts_tablet_manager.cc:1397] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260928 02:55:38.107437 10995 raft_consensus.cc:348] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.107934 10995 raft_consensus.cc:374] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:38.108068 10995 raft_consensus.cc:729] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad389a62e2d14f5899a1fbc97b795536, State: Initialized, Role: FOLLOWER
I20260928 02:55:38.108878 10998 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: No bootstrap required, opened a new log
I20260928 02:55:38.108745 10995 consensus_queue.cc:260] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.109236 10998 ts_tablet_manager.cc:1397] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20260928 02:55:38.113711 10996 raft_consensus.cc:348] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.114377 10996 raft_consensus.cc:374] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:38.114578 10996 raft_consensus.cc:729] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37468f93c61143a49e27aadf1258e984, State: Initialized, Role: FOLLOWER
I20260928 02:55:38.115278 10979 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:38.115360 10996 consensus_queue.cc:260] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.116557 10995 ts_tablet_manager.cc:1428] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Time spent starting tablet: real 0.020s user 0.016s sys 0.004s
I20260928 02:55:38.117643 10848 heartbeater.cc:503] Master 127.27.229.254:46197 was elected leader, sending a full tablet report...
I20260928 02:55:38.118247 10996 ts_tablet_manager.cc:1428] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
I20260928 02:55:38.119338 10998 raft_consensus.cc:348] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.119884 10998 raft_consensus.cc:374] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:38.120042 10998 raft_consensus.cc:729] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406a33188efa422e8ec38541b4a1b812, State: Initialized, Role: FOLLOWER
I20260928 02:55:38.120774 10998 consensus_queue.cc:260] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.122773 10998 ts_tablet_manager.cc:1428] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Time spent starting tablet: real 0.013s user 0.012s sys 0.003s
I20260928 02:55:38.205229 11001 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:38.205516 11001 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.207460 11001 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057)
I20260928 02:55:38.214345 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.207682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41937 (local address 127.27.229.195:39423)
0928 02:55:38.207889 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.207910 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:38.208051 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:38.208468 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:38.208478 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:38.208504 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:38.208732 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.208744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.210148 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.210152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.212122 (+ 1970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.212137 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.213708 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.213712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.213851 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.213875 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:55:38.213962 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:55:38.214072 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:55:38.214488 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.207849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:39423 (local address 127.27.229.194:41937)
0928 02:55:38.208075 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.208085 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.208110 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:55:38.208201 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.208210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.208312 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.208420 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:38.208903 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.209983 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.212299 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.213538 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.213713 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.213799 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.214011 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.214165 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:55:38.215114 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.208084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:56261 (local address 127.27.229.193:39057)
0928 02:55:38.208314 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.208321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.208779 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:55:38.208900 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.208906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.208995 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.209105 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:38.209641 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.210546 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.212787 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.214138 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.214309 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.214385 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.214744 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.214930 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:55:38.215659 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.207842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.27.229.195:56261)
0928 02:55:38.208638 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.208658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:38.208773 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:38.209159 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:38.209164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:38.209191 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:38.209464 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.209472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.210708 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.210713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.212625 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:38.212634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.214354 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.214363 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:38.214550 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.214586 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:55:38.215308 (+ 722us) client_negotiation.cc:240] Negotiation successful
0928 02:55:38.215422 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":440,"threads_started":1}
I20260928 02:55:38.215627 10803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ce2c1fc90343828734d7fbb91a57ac" candidate_uuid: "ad389a62e2d14f5899a1fbc97b795536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37468f93c61143a49e27aadf1258e984" is_pre_election: true
I20260928 02:55:38.216212 10803 raft_consensus.cc:2454] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad389a62e2d14f5899a1fbc97b795536 in term 0.
I20260928 02:55:38.216624 10671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ce2c1fc90343828734d7fbb91a57ac" candidate_uuid: "ad389a62e2d14f5899a1fbc97b795536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406a33188efa422e8ec38541b4a1b812" is_pre_election: true
I20260928 02:55:38.216971 10867 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536; no voters:
I20260928 02:55:38.217108 10671 raft_consensus.cc:2454] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad389a62e2d14f5899a1fbc97b795536 in term 0.
I20260928 02:55:38.217406 11001 raft_consensus.cc:2781] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:55:38.217572 11001 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:55:38.217729 11001 raft_consensus.cc:3037] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:38.220283 11001 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.221026 11001 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [CANDIDATE]: Term 1 election: Requested vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057)
I20260928 02:55:38.221553 10803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ce2c1fc90343828734d7fbb91a57ac" candidate_uuid: "ad389a62e2d14f5899a1fbc97b795536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37468f93c61143a49e27aadf1258e984"
I20260928 02:55:38.221562 10671 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50ce2c1fc90343828734d7fbb91a57ac" candidate_uuid: "ad389a62e2d14f5899a1fbc97b795536" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406a33188efa422e8ec38541b4a1b812"
I20260928 02:55:38.221786 10803 raft_consensus.cc:3037] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:38.221799 10671 raft_consensus.cc:3037] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:55:38.222606 10980 tablet.cc:2369] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:38.224381 10803 raft_consensus.cc:2454] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad389a62e2d14f5899a1fbc97b795536 in term 1.
I20260928 02:55:38.224545 10671 raft_consensus.cc:2454] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad389a62e2d14f5899a1fbc97b795536 in term 1.
I20260928 02:55:38.224893 10867 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536; no voters:
I20260928 02:55:38.225224 11001 raft_consensus.cc:2781] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:38.225739 11001 raft_consensus.cc:686] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 1 LEADER]: Becoming Leader. State: Replica: ad389a62e2d14f5899a1fbc97b795536, State: Running, Role: LEADER
I20260928 02:55:38.226213 11001 consensus_queue.cc:237] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:38.231066 10530 catalog_manager.cc:5495] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 reported cstate change: term changed from 0 to 1, leader changed from <none> to ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195). New cstate: current_term: 1 leader_uuid: "ad389a62e2d14f5899a1fbc97b795536" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } health_report { overall_health: UNKNOWN } } }
W20260928 02:55:38.250063 10849 tablet.cc:2369] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:55:38.276028 10717 tablet.cc:2369] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:55:38.281030 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.274499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56936 (local address 127.27.229.195:41015)
0928 02:55:38.274780 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.274788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.274810 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:55:38.274949 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.274953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.275045 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.275256 (+ 211us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:55:38.275645 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.276750 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.278448 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.279001 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.279119 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.279749 (+ 630us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:55:38.280324 (+ 575us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:55:38.280456 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.280582 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.280683 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 02:55:38.290462 10671 raft_consensus.cc:1261] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Refusing update from remote peer ad389a62e2d14f5899a1fbc97b795536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:38.290459 10803 raft_consensus.cc:1261] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Refusing update from remote peer ad389a62e2d14f5899a1fbc97b795536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:55:38.291576 11005 consensus_queue.cc:1035] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:38.291945 11001 consensus_queue.cc:1035] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:55:38.304713 11008 mvcc.cc:205] Tried to move back new op lower bound from 7334150710426804224 to 7334150710178451456. Current Snapshot: MvccSnapshot[applied={T|T < 7334150710426804224}]
I20260928 02:55:38.329699 10531 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:52640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:38.330057 10531 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:52640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:55:38.334074 10531 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Sending DeleteTablet for 3 replicas of tablet 50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:38.335460 10783 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:38 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:41210
I20260928 02:55:38.335530 10914 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:38 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56926
I20260928 02:55:38.335543 10651 tablet_service.cc:1514] Processing DeleteTablet for tablet 50ce2c1fc90343828734d7fbb91a57ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:55:38 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:59118
I20260928 02:55:38.336978 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10568
I20260928 02:55:38.350000 11019 tablet_replica.cc:330] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: stopping tablet replica
I20260928 02:55:38.350710 11019 raft_consensus.cc:2229] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:55:38.351838 11019 raft_consensus.cc:2258] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:38.353041 11018 tablet_replica.cc:330] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: stopping tablet replica
I20260928 02:55:38.353674 11017 tablet_replica.cc:330] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: stopping tablet replica
I20260928 02:55:38.354082 11018 raft_consensus.cc:2229] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:38.354293 11017 raft_consensus.cc:2229] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:55:38.354725 11018 raft_consensus.cc:2258] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:38.354791 11017 raft_consensus.cc:2258] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:55:38.357282 11019 ts_tablet_manager.cc:1905] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:38.359565 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.338084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55424 (local address 127.27.229.193:39057)
0928 02:55:38.338274 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.338280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.338301 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:55:38.338475 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.338481 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.338567 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.338678 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:38.341112 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.342361 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.345083 (+ 2722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.345787 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.345954 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.350535 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:38.350557 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:38.350578 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:38.350631 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:38.355603 (+ 4972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.356696 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.356706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.356710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.356892 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.357378 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.357384 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.357387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.357976 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:38.358322 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.358656 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.358950 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:55:38.359879 11017 ts_tablet_manager.cc:1905] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:38.360570 10711 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:38.367969 11017 ts_tablet_manager.cc:1918] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:38.368251 11017 log.cc:1196] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:38.368291 11019 ts_tablet_manager.cc:1918] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:55:38.368551 11019 log.cc:1196] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:38.369037 11017 ts_tablet_manager.cc:1939] T 50ce2c1fc90343828734d7fbb91a57ac P 37468f93c61143a49e27aadf1258e984: Deleting consensus metadata
I20260928 02:55:38.369283 11019 ts_tablet_manager.cc:1939] T 50ce2c1fc90343828734d7fbb91a57ac P ad389a62e2d14f5899a1fbc97b795536: Deleting consensus metadata
I20260928 02:55:38.371276 10517 catalog_manager.cc:4841] TS ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): tablet 50ce2c1fc90343828734d7fbb91a57ac (table test-table [id=428172a78fcb43ca82833c0c35d6b192]) successfully deleted
I20260928 02:55:38.371644 10515 catalog_manager.cc:4841] TS 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): tablet 50ce2c1fc90343828734d7fbb91a57ac (table test-table [id=428172a78fcb43ca82833c0c35d6b192]) successfully deleted
I20260928 02:55:38.360970 11018 ts_tablet_manager.cc:1905] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:55:38.463366 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10568
W20260928 02:55:38.485149 10517 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.229.193:39057 (error 108)
I20260928 02:55:38.485363 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10720
I20260928 02:55:38.487555 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.487097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55432)
0928 02:55:38.487280 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.487389 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 02:55:38.489737 10517 catalog_manager.cc:4584] TS 406a33188efa422e8ec38541b4a1b812 (127.27.229.193:39057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50ce2c1fc90343828734d7fbb91a57ac: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
I20260928 02:55:38.504913 10989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.486416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.27.229.194:41937)
0928 02:55:38.486603 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.486609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.486631 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:55:38.487315 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.487321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.487419 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.487519 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:38.489597 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.490713 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.492860 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.493499 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.493634 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.497248 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:38.497264 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:38.497285 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:38.497331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:38.501382 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.502277 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.502286 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.502290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.502442 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.502872 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.502878 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.502881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.503413 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:38.503711 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.504060 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.504320 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:55:38.505646 10843 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:38.543787 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.543301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55434)
0928 02:55:38.543516 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.543633 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:55:38.544857 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.544506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55450)
0928 02:55:38.544653 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.544750 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:55:38.605481 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.604911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55466)
0928 02:55:38.605168 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.605286 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:55:38.606710 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.606289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55478)
0928 02:55:38.606464 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.606566 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 02:55:38.630765 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10720
I20260928 02:55:38.649358 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10851
I20260928 02:55:38.665946 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.650278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40574 (local address 127.27.229.195:41015)
0928 02:55:38.650466 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:38.650471 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:38.650519 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:55:38.650756 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:38.650760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:38.650901 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:38.650989 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:38.652880 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.653742 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.655762 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:38.656232 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:38.656403 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:38.659362 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:38.659375 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:38.659390 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:38.659431 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:38.662760 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.663669 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.663683 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.663690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.663853 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:38.664274 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:38.664279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:38.664284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:38.664647 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:38.664984 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:38.665296 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:55:38.665530 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:55:38.666749 10974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:38.687681 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.687139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55494)
0928 02:55:38.687370 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.687470 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 02:55:38.688773 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.688357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55496)
0928 02:55:38.688556 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.688665 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 02:55:38.791843 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10851
I20260928 02:55:38.813865 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 10498
I20260928 02:55:38.815181 10561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:38.931389 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 10498
I20260928 02:55:38.931480 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:38.930883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39057 (local address 127.0.0.1:55512)
0928 02:55:38.931148 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:38.931273 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:39057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:55:38.953694 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--webserver_interface=127.27.229.254
--webserver_port=37513
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:39.326642 11025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:39.327246 11025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:39.327426 11025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:39.335141 11025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:39.335331 11025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:39.335443 11025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:39.335549 11025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:39.335636 11025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:39.346182 11025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:46197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=37513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:39.347762 11025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:39.349136 11025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:39.362231 11030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:39.362135 11031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:39.362200 11033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:39.363749 11025 server_base.cc:1034] running on GCE node
I20260928 02:55:39.365049 11025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:39.367107 11025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:39.368399 11025 hybrid_clock.cc:648] HybridClock initialized: now 1790564139368318 us; error 78 us; skew 500 ppm
I20260928 02:55:39.368921 11025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:39 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:39.373406 11025 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:39.375355 11025 webserver.cc:460] Webserver started at http://127.27.229.254:37513/ using document root <none> and password file <none>
I20260928 02:55:39.376135 11025 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:39.376295 11025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:39.382467 11025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 02:55:39.386045 11040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:39.388197 11025 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:55:39.388429 11025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "cb3cc013b5634f36bd62bc3561a04696"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "c33cbd0e678ef94bb33854303f397a7f"
server_key_iv: "a2de3bfad0e018ef3c05a93298871eda"
server_key_version: "encryptionkey@0"
I20260928 02:55:39.389547 11025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:39.408113 11025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:39.412979 11025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:39.413296 11025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:39.433493 11025 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:46197
I20260928 02:55:39.433501 11091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:46197 every 8 connection(s)
I20260928 02:55:39.435091 11025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:39.439186 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11025
I20260928 02:55:39.440022 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:39057
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=36485
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:39.442355 11092 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:55:39.445971 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap starting.
I20260928 02:55:39.477226 11092 log.cc:824] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:39.499001 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:55:39.499719 11092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696: Bootstrap complete.
I20260928 02:55:39.515734 11092 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:39.517120 11092 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Initialized, Role: FOLLOWER
I20260928 02:55:39.517905 11092 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:39.518215 11092 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:39.518343 11092 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:39.518532 11092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:55:39.525148 11092 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:39.525872 11092 leader_election.cc:302] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb3cc013b5634f36bd62bc3561a04696; no voters:
I20260928 02:55:39.526666 11092 leader_election.cc:288] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:55:39.526901 11096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:55:39.528223 11096 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [term 3 LEADER]: Becoming Leader. State: Replica: cb3cc013b5634f36bd62bc3561a04696, State: Running, Role: LEADER
I20260928 02:55:39.528849 11096 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } }
I20260928 02:55:39.529383 11092 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:39.531890 11097 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:39.532371 11097 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:39.533327 11098 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb3cc013b5634f36bd62bc3561a04696. Latest consensus state: current_term: 3 leader_uuid: "cb3cc013b5634f36bd62bc3561a04696" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb3cc013b5634f36bd62bc3561a04696" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 46197 } } }
I20260928 02:55:39.533624 11098 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb3cc013b5634f36bd62bc3561a04696 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:39.535265 11102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:39.546895 11102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:39.549289 11102 catalog_manager.cc:1259] Loaded cluster ID: 12715fb200b24911865280ca0b18e6b9
I20260928 02:55:39.549467 11102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 02:55:39.552415 11102 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2dfd83f9ad google::LogMessage::Fail() at ??:0
@ 0x7f2dfd84288a google::LogMessage::SendToLog() at ??:0
@ 0x7f2dfd83f4de google::LogMessage::Flush() at ??:0
@ 0x7f2dfd840a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2e13272512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f2e1326f976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2e132f7610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f2e132f7439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2e0048b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2e0048e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f2e0048e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f2e00469eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2e037436db start_thread at ??:0
@ 0x7f2dfc6d961f clone at ??:0
W20260928 02:55:39.850696 11094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:39.851104 11094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:39.851293 11094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:39.859064 11094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:39.859227 11094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:39.859401 11094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:39.869952 11094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:39057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=36485
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:39.871361 11094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:39.872684 11094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:39.886409 11136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:39.886451 11133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:39.886497 11134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:39.887264 11094 server_base.cc:1034] running on GCE node
I20260928 02:55:39.887956 11094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:39.888993 11094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:39.890244 11094 hybrid_clock.cc:648] HybridClock initialized: now 1790564139890179 us; error 60 us; skew 500 ppm
I20260928 02:55:39.890578 11094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:39 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:39.894395 11094 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:39.895977 11094 webserver.cc:460] Webserver started at http://127.27.229.193:36485/ using document root <none> and password file <none>
I20260928 02:55:39.896641 11094 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:39.896749 11094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:39.902218 11094 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:55:39.905407 11143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:39.907104 11094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:55:39.907271 11094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "406a33188efa422e8ec38541b4a1b812"
format_stamp: "Formatted at 2026-09-28 02:55:33 on dist-test-slave-s1kg"
server_key: "d5e3b2ea865bd51be97ac31714773631"
server_key_iv: "513110edede80db64a8c132125d215e5"
server_key_version: "encryptionkey@0"
I20260928 02:55:39.908249 11094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:39.928146 11094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:39.933038 11094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:39.933382 11094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:39.934489 11094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:39.936370 11150 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 02:55:39.942291 11094 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:55:39.942420 11094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260928 02:55:39.942606 11094 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:55:39.945979 11094 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:55:39.946079 11094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 02:55:39.946487 11150 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Bootstrap starting.
I20260928 02:55:39.955888 11150 log.cc:824] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:39.972580 11150 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:55:39.973511 11150 tablet_bootstrap.cc:492] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Bootstrap complete.
I20260928 02:55:39.974139 11150 ts_tablet_manager.cc:1397] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20260928 02:55:39.986146 11094 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39057
I20260928 02:55:39.985148 11150 raft_consensus.cc:348] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:39.986356 11150 raft_consensus.cc:729] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406a33188efa422e8ec38541b4a1b812, State: Initialized, Role: FOLLOWER
I20260928 02:55:39.986172 11257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39057 every 8 connection(s)
I20260928 02:55:39.987035 11150 consensus_queue.cc:260] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:39.987864 11094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:39.988564 11150 ts_tablet_manager.cc:1428] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812: Time spent starting tablet: real 0.014s user 0.003s sys 0.011s
I20260928 02:55:39.988960 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11094
I20260928 02:55:39.989650 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:41937
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=46873
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:55:39.993569 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:39.991579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:34805)
0928 02:55:39.992484 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:39.992759 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":419,"threads_started":1}
W20260928 02:55:39.996168 11259 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:46197 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:46197: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
I20260928 02:55:39.997248 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:39.996776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:53765)
0928 02:55:39.996989 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:39.997092 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 02:55:39.998407 11261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:39.997982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:59743)
0928 02:55:39.998166 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:39.998263 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260928 02:55:39.998757 11259 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:55:40.332891 11262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:40.333284 11262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:40.333444 11262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:40.340822 11262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:40.340945 11262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:40.341125 11262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:40.351143 11262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:41937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=46873
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:40.352465 11262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:40.353878 11262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:40.367741 11268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:40.367882 11270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:40.367806 11267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:40.368391 11262 server_base.cc:1034] running on GCE node
I20260928 02:55:40.369210 11262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:40.370201 11262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:40.371444 11262 hybrid_clock.cc:648] HybridClock initialized: now 1790564140371382 us; error 59 us; skew 500 ppm
I20260928 02:55:40.371793 11262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:40 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564140, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:40.375211 11262 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:40.376977 11262 webserver.cc:460] Webserver started at http://127.27.229.194:46873/ using document root <none> and password file <none>
I20260928 02:55:40.377732 11262 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:40.377887 11262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:40.383898 11262 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:55:40.387406 11277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.388960 11262 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 02:55:40.389230 11262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "37468f93c61143a49e27aadf1258e984"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "85372e2533ae1f017612d7f27a827182"
server_key_iv: "c4859c89fc46d6ecc1d6319ebb717bc5"
server_key_version: "encryptionkey@0"
I20260928 02:55:40.390457 11262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:40.409195 11262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:40.415702 11262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:40.416016 11262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:40.417196 11262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:40.418672 11262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:40.418823 11262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.419021 11262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:40.419124 11262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.455355 11262 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41937
I20260928 02:55:40.455411 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41937 every 8 connection(s)
I20260928 02:55:40.456844 11262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:40.463388 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.460934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:35739)
0928 02:55:40.461965 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.462249 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":433,"threads_started":1}
W20260928 02:55:40.464242 11390 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:46197 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:46197: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
I20260928 02:55:40.465407 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.464994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:53947)
0928 02:55:40.465205 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.465292 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:55:40.465937 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11262
I20260928 02:55:40.466323 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.465950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:41269)
0928 02:55:40.466097 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.466188 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:55:40.466566 11390 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:55:40.466640 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:41015
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:46197
--webserver_port=35239
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:37993
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:40.799846 11393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:40.800133 11393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:40.800309 11393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:40.807799 11393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:40.807927 11393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:40.808104 11393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:40.818359 11393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:37993
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:41015
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=35239
--tserver_master_addrs=127.27.229.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:40.819772 11393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:40.821102 11393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:40.835052 11398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:40.835304 11401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:40.835915 11399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:40.836387 11393 server_base.cc:1034] running on GCE node
I20260928 02:55:40.837092 11393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:40.838161 11393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:40.839418 11393 hybrid_clock.cc:648] HybridClock initialized: now 1790564140839363 us; error 52 us; skew 500 ppm
I20260928 02:55:40.839823 11393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:40 dist-test-slave-s1kg krb5kdc[9946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564140, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:40.843179 11393 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:40.844748 11393 webserver.cc:460] Webserver started at http://127.27.229.195:35239/ using document root <none> and password file <none>
I20260928 02:55:40.845458 11393 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:40.845615 11393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:40.851536 11393 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:55:40.855158 11408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.856855 11393 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:55:40.857093 11393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "ad389a62e2d14f5899a1fbc97b795536"
format_stamp: "Formatted at 2026-09-28 02:55:34 on dist-test-slave-s1kg"
server_key: "8628207532c6c2b21b40cb4fc4925752"
server_key_iv: "fa3528ace73581e09dfd74c9cec1de93"
server_key_version: "encryptionkey@0"
I20260928 02:55:40.858196 11393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:40.902454 11393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:40.907148 11393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:40.907444 11393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:40.908483 11393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:40.909937 11393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:40.910064 11393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.910200 11393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:40.910286 11393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:40.945757 11393 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:41015
I20260928 02:55:40.945844 11520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:41015 every 8 connection(s)
I20260928 02:55:40.947218 11393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:40.952111 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11393
I20260928 02:55:40.952296 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.950271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:47739)
0928 02:55:40.951189 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.951421 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":307,"threads_started":1}
W20260928 02:55:40.953035 11521 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:46197 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:46197: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
I20260928 02:55:40.953923 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.953538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:33187)
0928 02:55:40.953689 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.953767 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:55:40.954819 11523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.954525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:48671)
0928 02:55:40.954643 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.954713 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:55:40.955147 11521 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:55:41.000072 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:40.999289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:33637)
0928 02:55:40.999749 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:40.999840 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":232,"threads_started":1}
I20260928 02:55:41.405381 11526 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:41.405761 11526 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:41.407612 11526 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
Sep 28 02:55:41 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
Sep 28 02:55:41 dist-test-slave-s1kg krb5kdc[9946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564139, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:55:41.433141 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.408208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:49287 (local address 127.27.229.194:41937)
0928 02:55:41.410435 (+ 2227us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:41.410448 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:41.410487 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:55:41.412702 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:41.412713 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:41.413225 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:41.413618 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:41.416786 (+ 3168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.418338 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.420308 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.420940 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.421126 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:41.425728 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:41.425735 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:41.425745 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:41.425787 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:41.429305 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430169 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:41.430175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:41.430180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:41.430371 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430760 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:41.430766 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:41.430769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:41.431371 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:41.431594 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:41.432343 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 02:55:41.432720 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1985,"thread_start_us":1845,"threads_started":1}
I20260928 02:55:41.433156 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.408278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48945 (local address 127.27.229.195:41015)
0928 02:55:41.410089 (+ 1811us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:41.410107 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:41.410168 (+ 61us) server_negotiation.cc:407] Connection header received
0928 02:55:41.412643 (+ 2475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:41.412657 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:41.413217 (+ 560us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:41.413622 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:41.416794 (+ 3172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.418371 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.420275 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.420960 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.421179 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:41.425439 (+ 4260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:41.425453 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:41.425476 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:41.425532 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:41.429186 (+ 3654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430071 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:41.430082 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:41.430091 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:41.430272 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430788 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:41.430792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:41.430794 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:41.431376 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:41.431589 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:41.432338 (+ 749us) server_negotiation.cc:299] Negotiation successful
0928 02:55:41.432720 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1600,"thread_start_us":1482,"threads_started":1}
I20260928 02:55:41.433168 11525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.407908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41937 (local address 127.27.229.193:49287)
0928 02:55:41.408223 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:41.408272 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:41.408921 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:41.413894 (+ 4973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:41.413907 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:41.415127 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:41.416488 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:41.416547 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.418560 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.418568 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:41.420106 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:41.420132 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.421178 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.421187 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:41.421470 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:41.422324 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:41.422357 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:41.425403 (+ 3046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:41.429465 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:41.429470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:41.429473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:41.430024 (+ 551us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:41.430504 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:41.430509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:41.430637 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:41.431575 (+ 938us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:41.431579 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:41.431835 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 02:55:41.432099 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 02:55:41.432707 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":202,"spinlock_wait_cycles":36224}
I20260928 02:55:41.433310 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.407978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:41015 (local address 127.27.229.193:48945)
0928 02:55:41.409764 (+ 1786us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:41.409785 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:41.409898 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:41.413894 (+ 3996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:41.413907 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:41.415127 (+ 1220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:41.416488 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:41.416522 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.418571 (+ 2049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.418575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:41.420112 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:41.420132 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:41.421198 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:41.421204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:41.421468 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:41.422370 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:41.422387 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:41.425099 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:41.429340 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:41.429352 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:41.429370 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:41.429909 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:41.430410 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:41.430418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:41.430424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:41.430635 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:41.431517 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:41.431526 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:41.431837 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 02:55:41.432099 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 02:55:41.432891 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1637,"mutex_wait_us":610,"spinlock_wait_cycles":30592,"thread_start_us":1478,"threads_started":1}
W20260928 02:55:41.436700 11144 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
W20260928 02:55:41.437618 11146 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:41.437740 11146 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 406a33188efa422e8ec38541b4a1b812; no voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536
I20260928 02:55:41.438100 11526 raft_consensus.cc:2726] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:55:41.468062 11530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.467156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:58313)
0928 02:55:41.467665 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:41.467808 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
I20260928 02:55:41.956455 11531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:41.955699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:44533)
0928 02:55:41.956142 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:41.956258 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":211,"threads_started":1}
I20260928 02:55:42.001804 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:42.001081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:44273)
0928 02:55:42.001525 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:42.001640 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":222,"threads_started":1}
I20260928 02:55:42.469656 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:42.468969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:51031)
0928 02:55:42.469410 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:42.469489 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":228,"threads_started":1}
I20260928 02:55:42.958207 11534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:42.957431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:36759)
0928 02:55:42.957889 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:42.958000 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":229,"threads_started":1}
I20260928 02:55:43.003698 11535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:43.002929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:43373)
0928 02:55:43.003436 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:43.003520 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260928 02:55:43.246351 11536 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:43.246552 11536 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:43.247259 11536 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
W20260928 02:55:43.248271 11146 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
W20260928 02:55:43.248433 11144 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:43.248554 11144 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 406a33188efa422e8ec38541b4a1b812; no voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536
I20260928 02:55:43.248821 11536 raft_consensus.cc:2726] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:55:43.471611 11537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:43.470640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:50897)
0928 02:55:43.471274 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:43.471385 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":342,"threads_started":1}
I20260928 02:55:43.960086 11538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:43.959264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:59149)
0928 02:55:43.959727 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:43.959828 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260928 02:55:44.005424 11539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:44.004728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:56899)
0928 02:55:44.005162 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:44.005256 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":231,"threads_started":1}
I20260928 02:55:44.473382 11540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:44.472612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:56125)
0928 02:55:44.473109 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:44.473198 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":254,"threads_started":1}
I20260928 02:55:44.961771 11541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:44.961030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:37467)
0928 02:55:44.961505 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:44.961595 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":237,"threads_started":1}
I20260928 02:55:45.007330 11542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:45.006456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:44475)
0928 02:55:45.007034 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:45.007126 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":258,"threads_started":1}
I20260928 02:55:45.475208 11543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:45.474389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:48645)
0928 02:55:45.474904 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:45.474994 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":231,"threads_started":1}
I20260928 02:55:45.963828 11544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:45.963062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:43027)
0928 02:55:45.963559 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:45.963654 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":262,"threads_started":1}
I20260928 02:55:46.009110 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:46.008355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:58795)
0928 02:55:46.008819 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:46.008916 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":242,"threads_started":1}
I20260928 02:55:46.399186 11546 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:46.399436 11546 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:46.400125 11546 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
W20260928 02:55:46.401237 11144 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
W20260928 02:55:46.401420 11146 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:46.401599 11146 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 406a33188efa422e8ec38541b4a1b812; no voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536
I20260928 02:55:46.401890 11546 raft_consensus.cc:2726] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:55:46.482455 11547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:46.481612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:49671)
0928 02:55:46.482188 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:46.482279 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":316,"threads_started":1}
I20260928 02:55:46.965812 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:46.964887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:55799)
0928 02:55:46.965453 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:46.965592 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":307,"threads_started":1}
I20260928 02:55:47.011178 11549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:47.010216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:44443)
0928 02:55:47.010861 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:47.010962 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":321,"threads_started":1}
I20260928 02:55:47.484508 11550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:47.483617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:38545)
0928 02:55:47.484181 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:47.484293 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":265,"threads_started":1}
I20260928 02:55:47.967679 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:47.966825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:42611)
0928 02:55:47.967396 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:47.967481 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":251,"threads_started":1}
I20260928 02:55:48.013267 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:48.012389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:48749)
0928 02:55:48.013001 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:48.013089 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":295,"threads_started":1}
I20260928 02:55:48.486495 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:48.485711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:35961)
0928 02:55:48.486215 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:48.486308 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":286,"threads_started":1}
I20260928 02:55:48.969549 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:48.968697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:42761)
0928 02:55:48.969265 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:48.969358 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":338,"threads_started":1}
I20260928 02:55:49.015125 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:49.014251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:43903)
0928 02:55:49.014777 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:49.014935 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":260,"threads_started":1}
I20260928 02:55:49.488271 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:49.487528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:32803)
0928 02:55:49.488031 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:49.488116 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":225,"threads_started":1}
I20260928 02:55:49.971686 11557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:49.970823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:52013)
0928 02:55:49.971416 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:49.971506 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":285,"threads_started":1}
I20260928 02:55:50.017051 11558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:50.016277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:34491)
0928 02:55:50.016796 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:50.016887 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":255,"threads_started":1}
I20260928 02:55:50.490032 11559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:50.489307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:44577)
0928 02:55:50.489766 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:50.489859 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":245,"threads_started":1}
I20260928 02:55:50.850497 11560 raft_consensus.cc:482] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:55:50.850682 11560 raft_consensus.cc:504] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37468f93c61143a49e27aadf1258e984" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41937 } } peers { permanent_uuid: "ad389a62e2d14f5899a1fbc97b795536" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 41015 } } peers { permanent_uuid: "406a33188efa422e8ec38541b4a1b812" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39057 } }
I20260928 02:55:50.851457 11560 leader_election.cc:288] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937), ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015)
W20260928 02:55:50.852399 11144 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 37468f93c61143a49e27aadf1258e984 (127.27.229.194:41937): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
W20260928 02:55:50.852592 11146 leader_election.cc:341] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ad389a62e2d14f5899a1fbc97b795536 (127.27.229.195:41015): Not found: Tablet not found: 50ce2c1fc90343828734d7fbb91a57ac
I20260928 02:55:50.852716 11146 leader_election.cc:302] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 406a33188efa422e8ec38541b4a1b812; no voters: 37468f93c61143a49e27aadf1258e984, ad389a62e2d14f5899a1fbc97b795536
I20260928 02:55:50.852929 11560 raft_consensus.cc:2726] T 50ce2c1fc90343828734d7fbb91a57ac P 406a33188efa422e8ec38541b4a1b812 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:55:50.973474 11561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:50.972699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:41557)
0928 02:55:50.973202 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:50.973290 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":269,"threads_started":1}
I20260928 02:55:51.018955 11562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:51.018010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:41153)
0928 02:55:51.018555 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:51.018679 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":262,"threads_started":1}
I20260928 02:55:51.491858 11563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:51.491088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:53507)
0928 02:55:51.491597 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:51.491690 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":251,"threads_started":1}
I20260928 02:55:51.975517 11564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:51.974653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:47671)
0928 02:55:51.975271 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:51.975366 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":345,"threads_started":1}
I20260928 02:55:52.020962 11565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:52.020102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:60439)
0928 02:55:52.020696 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:52.020801 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":340,"threads_started":1}
I20260928 02:55:52.493741 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:52.492912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:52461)
0928 02:55:52.493446 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:52.493550 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":277,"threads_started":1}
I20260928 02:55:52.977218 11567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:52.976471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:54967)
0928 02:55:52.976944 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:52.977028 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260928 02:55:53.022792 11568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:53.021979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:54135)
0928 02:55:53.022510 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:53.022608 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":315,"threads_started":1}
I20260928 02:55:53.495802 11569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:53.494871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:43203)
0928 02:55:53.495492 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:53.495638 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":320,"threads_started":1}
I20260928 02:55:53.979122 11570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:53.978248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:34195)
0928 02:55:53.978741 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:53.978904 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":269,"threads_started":1}
I20260928 02:55:54.024713 11571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:54.023857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:58309)
0928 02:55:54.024426 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:54.024530 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":337,"threads_started":1}
I20260928 02:55:54.497532 11572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:54.496740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:55951)
0928 02:55:54.497245 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:54.497334 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260928 02:55:54.980923 11573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:54.980175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:53915)
0928 02:55:54.980626 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:54.980731 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":227,"threads_started":1}
I20260928 02:55:55.026984 11574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:55.026150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.193:44567)
0928 02:55:55.026625 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:55.026719 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":258,"threads_started":1}
I20260928 02:55:55.499679 11575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:55.498945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.194:58367)
0928 02:55:55.499388 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:55.499516 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260928 02:55:55.957522 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11094
I20260928 02:55:55.980391 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:55.958865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53660 (local address 127.27.229.193:39057)
0928 02:55:55.959451 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:55.959459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:55.959510 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:55:55.959606 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:55.959615 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:55.959966 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:55.960180 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:55.961854 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:55.963179 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:55.964555 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:55.965153 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:55.965267 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:55.973976 (+ 8709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:55.973994 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:55.974010 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:55.974070 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:55.977253 (+ 3183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:55.978095 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:55.978102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:55.978105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:55.978249 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:55.978656 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:55.978660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:55.978662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:55.979175 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:55.979428 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:55.979742 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 02:55:55.980029 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":267,"threads_started":1}
I20260928 02:55:55.981714 11253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:55.982976 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:55.982044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:46197 (local address 127.27.229.195:38659)
0928 02:55:55.982597 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:55.982746 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:46197: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":253,"threads_started":1}
I20260928 02:55:56.073405 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11094
I20260928 02:55:56.089323 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11262
I20260928 02:55:56.104884 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:56.090156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41944 (local address 127.27.229.194:41937)
0928 02:55:56.090652 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:56.090656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:56.090673 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:55:56.090727 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:56.090731 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:56.090788 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:56.090945 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:56.092438 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:56.093286 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:56.094647 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:56.095221 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:56.095369 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:56.098584 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:56.098590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:56.098593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:56.098632 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:56.101707 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:56.102531 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:56.102536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:56.102539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:56.102715 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:56.103391 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:56.103396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:56.103399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:56.103726 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:56.103931 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:56.104364 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 02:55:56.104561 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
I20260928 02:55:56.105684 11385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:56.191792 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11262
I20260928 02:55:56.209728 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11393
I20260928 02:55:56.223817 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:56.210533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.27.229.195:41015)
0928 02:55:56.211159 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:56.211163 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:56.211188 (+ 25us) server_negotiation.cc:407] Connection header received
0928 02:55:56.211244 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:56.211248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:56.211308 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:56.211404 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:56.212818 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:56.213656 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:56.215138 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:56.215581 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:56.215683 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:56.218236 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:56.218241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:56.218244 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:56.218277 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:56.220981 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:56.221666 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:56.221670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:56.221672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:56.221818 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:56.222210 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:56.222213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:56.222216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:56.222494 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:56.222726 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:56.223200 (+ 474us) server_negotiation.cc:299] Negotiation successful
0928 02:55:56.223374 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":347,"threads_started":1}
I20260928 02:55:56.224534 11516 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:55:56.308058 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11393
2026-09-28T02:55:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24781 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:55:56 dist-test-slave-s1kg krb5kdc[11585](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:55:56 dist-test-slave-s1kg krb5kdc[11585](info): set up 2 sockets
Sep 28 02:55:56 dist-test-slave-s1kg krb5kdc[11585](info): commencing operation
krb5kdc: starting...
W20260928 02:55:57.431835 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:55:57 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:55:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:55:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:55:57.678754 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:58.019476 11598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:58.019799 11598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:58.019966 11598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:58.027489 11598 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:55:58.027613 11598 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:55:58.027652 11598 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:55:58.027752 11598 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:55:58.027839 11598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:58.027932 11598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:58.028003 11598 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:55:58.028093 11598 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:55:58.038659 11598 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:58.040050 11598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:58.041393 11598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:58.054813 11603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:58.054771 11606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:58.054729 11604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:58.055105 11598 server_base.cc:1034] running on GCE node
I20260928 02:55:58.056110 11598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:58.057585 11598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:58.058892 11598 hybrid_clock.cc:648] HybridClock initialized: now 1790564158058773 us; error 37 us; skew 500 ppm
I20260928 02:55:58.059324 11598 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:58 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564158, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:58.062821 11598 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:55:58.064603 11598 webserver.cc:460] Webserver started at http://127.27.229.254:42795/ using document root <none> and password file <none>
I20260928 02:55:58.065423 11598 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:58.065572 11598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:58.065922 11598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:58.069425 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "c9de88825dbf4554b6ae456623a68c20"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "799327b22bb4ef338fc4af846bde8253"
server_key_iv: "3282bb88e8d1a2cf5959d49191a6326b"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.070348 11598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "c9de88825dbf4554b6ae456623a68c20"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "799327b22bb4ef338fc4af846bde8253"
server_key_iv: "3282bb88e8d1a2cf5959d49191a6326b"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.076400 11598 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260928 02:55:58.080228 11613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:58.081818 11598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:55:58.082058 11598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "c9de88825dbf4554b6ae456623a68c20"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "799327b22bb4ef338fc4af846bde8253"
server_key_iv: "3282bb88e8d1a2cf5959d49191a6326b"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.082330 11598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:58.102650 11598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:58.107872 11598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:58.108199 11598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:58.129184 11598 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:35261
I20260928 02:55:58.129197 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:35261 every 8 connection(s)
I20260928 02:55:58.130715 11598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:55:58.133277 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11598
I20260928 02:55:58.133514 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:55:58.134349 28567 external_mini_cluster.cc:1428] Setting key 53b90d98019ec519a5ee85ae41f4a879
I20260928 02:55:58.135380 11665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:58.150494 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap starting.
Sep 28 02:55:58 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:58.154587 11665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:58.156059 11665 log.cc:824] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:58.159895 11665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: No bootstrap required, opened a new log
I20260928 02:55:58.161599 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:58.136589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50780 (local address 127.27.229.254:35261)
0928 02:55:58.137406 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:58.137418 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:58.137458 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:55:58.139967 (+ 2509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:58.139990 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:58.140439 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:58.140775 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:58.142641 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.143991 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.145870 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.146590 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.146743 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:58.151996 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:58.152012 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:58.152061 (+ 49us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:58.152135 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:58.156565 (+ 4430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:58.157658 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:58.157665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:58.157672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:58.157907 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:58.158323 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:58.158327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:58.158330 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:58.159074 (+ 744us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:58.159375 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:58.159784 (+ 409us) server_negotiation.cc:299] Negotiation successful
0928 02:55:58.160230 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":367,"threads_started":1}
I20260928 02:55:58.168699 11665 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:55:58.169080 11665 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:55:58.169144 11665 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Initialized, Role: FOLLOWER
I20260928 02:55:58.169700 11665 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:55:58.169930 11665 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:55:58.170033 11665 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:55:58.170151 11665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:55:58.172398 11665 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:55:58.172866 11665 leader_election.cc:302] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9de88825dbf4554b6ae456623a68c20; no voters:
I20260928 02:55:58.173432 11665 leader_election.cc:288] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:55:58.173614 11670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:55:58.175024 11670 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 LEADER]: Becoming Leader. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Running, Role: LEADER
I20260928 02:55:58.175712 11670 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:55:58.175828 11665 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:55:58.179035 11671 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:55:58.178993 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9de88825dbf4554b6ae456623a68c20. Latest consensus state: current_term: 1 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:55:58.179450 11671 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:58.179455 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:55:58.180451 11679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:55:58.187395 11679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:55:58.196430 11679 catalog_manager.cc:1347] Generated new cluster ID: 58f040dd68014adc8713415808340d6f
I20260928 02:55:58.196556 11679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:55:58.215698 11679 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:55:58.217103 11679 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:55:58.228892 11679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Generated new TSK 0
I20260928 02:55:58.229812 11679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:55:58.307086 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:58.651525 11691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:58.651805 11691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:58.651899 11691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:58.659247 11691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:58.659363 11691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:58.659478 11691 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:55:58.669318 11691 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:58.670601 11691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:58.671950 11691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:58.685272 11696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:58.685160 11697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:58.685240 11699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:58.686203 11691 server_base.cc:1034] running on GCE node
I20260928 02:55:58.686904 11691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:58.688004 11691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:58.689229 11691 hybrid_clock.cc:648] HybridClock initialized: now 1790564158689141 us; error 62 us; skew 500 ppm
I20260928 02:55:58.689567 11691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:58 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564158, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:58.692884 11691 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:55:58.694387 11691 webserver.cc:460] Webserver started at http://127.27.229.193:42459/ using document root <none> and password file <none>
I20260928 02:55:58.695180 11691 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:58.695329 11691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:58.695642 11691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:58.698719 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "a18d53795922bcd0ab0b182d42c06ea7"
server_key_iv: "755b2fdfd755041770604bbd3fbc8d35"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.699654 11691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "a18d53795922bcd0ab0b182d42c06ea7"
server_key_iv: "755b2fdfd755041770604bbd3fbc8d35"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.705380 11691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:55:58.709391 11706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:58.710912 11691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:55:58.711128 11691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "a18d53795922bcd0ab0b182d42c06ea7"
server_key_iv: "755b2fdfd755041770604bbd3fbc8d35"
server_key_version: "encryptionkey@0"
I20260928 02:55:58.711337 11691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:58.737180 11691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:58.741773 11691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:58.742070 11691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:58.743139 11691 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:58.744537 11691 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:58.744637 11691 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:58.744776 11691 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:58.744843 11691 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:58.780268 11691 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:36789
I20260928 02:55:58.780349 11818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:36789 every 8 connection(s)
I20260928 02:55:58.781579 11691 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:55:58.782191 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11691
I20260928 02:55:58.782449 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:55:58.783291 28567 external_mini_cluster.cc:1428] Setting key 8ba77953730896fa8121320768ea448d
Sep 28 02:55:58 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564158, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:58.805035 11821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:58.784471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:47583 (local address 127.27.229.254:35261)
0928 02:55:58.785585 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:58.785591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:58.786246 (+ 655us) server_negotiation.cc:407] Connection header received
0928 02:55:58.788256 (+ 2010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:58.788264 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:58.788354 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:58.788470 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:58.789911 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.791124 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.792736 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.793182 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.793286 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:58.797552 (+ 4266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:58.797561 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:58.797577 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:58.797628 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:58.801355 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:58.802219 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:58.802227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:58.802233 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:58.802409 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:58.802873 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:58.802881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:58.802887 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:58.803296 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:58.803461 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:58.804245 (+ 784us) server_negotiation.cc:299] Negotiation successful
0928 02:55:58.804522 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":854,"threads_started":1}
I20260928 02:55:58.805869 11822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:58.785137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:47583)
0928 02:55:58.786086 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:58.786131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:58.787981 (+ 1850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:58.788661 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:58.788669 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:58.789174 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:58.789732 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:58.789747 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.791295 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.791299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:58.792585 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:58.792597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:58.793436 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:58.793442 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:58.793791 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:58.794582 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:58.794605 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:58.797292 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:58.801537 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:58.801544 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:58.801568 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:58.802077 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:58.802526 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:58.802530 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:58.802532 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:58.802741 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:58.803443 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:58.803449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:58.803834 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 02:55:58.804106 (+ 272us) client_negotiation.cc:240] Negotiation successful
0928 02:55:58.804540 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":417,"threads_started":1}
I20260928 02:55:58.807597 11819 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:55:58.807930 11819 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:58.808688 11819 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:55:58.811091 11630 ts_manager.cc:194] Registered new tserver with Master: 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
I20260928 02:55:58.812657 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:47583
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:55:58.841727 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:59.187362 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:59.187793 11825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:59.187970 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:59.200731 11825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:59.200909 11825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:59.201094 11825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:55:59.217510 11825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:59.218946 11825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:59.220291 11825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:59.234495 11831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:59.234532 11830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:59.234532 11833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:59.235718 11825 server_base.cc:1034] running on GCE node
I20260928 02:55:59.236459 11825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:59.237499 11825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:59.238760 11825 hybrid_clock.cc:648] HybridClock initialized: now 1790564159238716 us; error 37 us; skew 500 ppm
I20260928 02:55:59.239216 11825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:59 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564159, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:59.242393 11825 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:55:59.244059 11825 webserver.cc:460] Webserver started at http://127.27.229.194:36233/ using document root <none> and password file <none>
I20260928 02:55:59.244825 11825 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:59.244930 11825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:59.245226 11825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:59.249331 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "324ec59257b9449a86ec5ce128f248c0"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "d0366868a69d78819d04f15e6555bfcd"
server_key_iv: "3478b1b64c48c933ff5aac35b86a1c64"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.250408 11825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "324ec59257b9449a86ec5ce128f248c0"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "d0366868a69d78819d04f15e6555bfcd"
server_key_iv: "3478b1b64c48c933ff5aac35b86a1c64"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.257683 11825 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260928 02:55:59.261760 11840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.263428 11825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:55:59.263638 11825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "324ec59257b9449a86ec5ce128f248c0"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "d0366868a69d78819d04f15e6555bfcd"
server_key_iv: "3478b1b64c48c933ff5aac35b86a1c64"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.263857 11825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:59.290230 11825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:59.295019 11825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:59.295368 11825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:59.296367 11825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:59.297715 11825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:59.297792 11825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.297880 11825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:59.297924 11825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.333257 11825 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:42687
I20260928 02:55:59.333353 11952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:42687 every 8 connection(s)
I20260928 02:55:59.334617 11825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:55:59.336917 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11825
I20260928 02:55:59.337105 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:55:59.337587 28567 external_mini_cluster.cc:1428] Setting key fa1c42428cb752abb72edb744f7f95e7
Sep 28 02:55:59 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564159, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:59.358270 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.337817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:47191 (local address 127.27.229.254:35261)
0928 02:55:59.338326 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.338334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.341514 (+ 3180us) server_negotiation.cc:407] Connection header received
0928 02:55:59.341569 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.341572 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.341641 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.341727 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:59.344061 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.344819 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.347447 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.347985 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.348149 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.352210 (+ 4061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:59.352216 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:59.352221 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:59.352257 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:59.355033 (+ 2776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.355844 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.355848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.355850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.355976 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.356450 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.356454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.356456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.356734 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:59.356888 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.357633 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.357895 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":236,"threads_started":1}
I20260928 02:55:59.359623 11956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.338477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:47191)
0928 02:55:59.339438 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:59.339497 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:59.341225 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:59.342058 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:59.342073 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:59.342967 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:59.343814 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.343848 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.345005 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.345014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.347262 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.347281 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.348255 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.348265 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.348590 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.349386 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:59.349410 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:59.351926 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:59.355193 (+ 3267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:59.355203 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:59.355231 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:59.355690 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:59.356094 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:59.356099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:59.356103 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:59.356318 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:59.356863 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:59.356871 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:59.357250 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 02:55:59.357531 (+ 281us) client_negotiation.cc:240] Negotiation successful
0928 02:55:59.358102 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":356,"threads_started":1}
I20260928 02:55:59.361756 11953 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:55:59.362119 11953 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:59.363112 11953 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:55:59.365489 11630 ts_manager.cc:194] Registered new tserver with Master: 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687)
I20260928 02:55:59.366432 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:47191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:55:59.399046 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:55:59.733705 11959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:55:59.734073 11959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:55:59.734261 11959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:55:59.741849 11959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:55:59.742025 11959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:55:59.742226 11959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:55:59.752368 11959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:55:59.753727 11959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:55:59.755050 11959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:55:59.769953 11965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:59.770004 11967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:55:59.770004 11964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:55:59.770920 11959 server_base.cc:1034] running on GCE node
I20260928 02:55:59.771663 11959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:55:59.772682 11959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:55:59.773939 11959 hybrid_clock.cc:648] HybridClock initialized: now 1790564159773882 us; error 50 us; skew 500 ppm
I20260928 02:55:59.774328 11959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:55:59 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564159, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:55:59.777961 11959 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:55:59.779627 11959 webserver.cc:460] Webserver started at http://127.27.229.195:46311/ using document root <none> and password file <none>
I20260928 02:55:59.780514 11959 fs_manager.cc:359] Metadata directory not provided
I20260928 02:55:59.780658 11959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:55:59.781033 11959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:55:59.785048 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "e151992d0192a90e17edc618a5b83812"
server_key_iv: "650f7fedb5020b5a3a75e112a6eaebc6"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.786104 11959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "e151992d0192a90e17edc618a5b83812"
server_key_iv: "650f7fedb5020b5a3a75e112a6eaebc6"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.791682 11959 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:55:59.795375 11974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.796803 11959 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:55:59.796994 11959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "e151992d0192a90e17edc618a5b83812"
server_key_iv: "650f7fedb5020b5a3a75e112a6eaebc6"
server_key_version: "encryptionkey@0"
I20260928 02:55:59.797215 11959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:55:59.811599 11959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:55:59.816004 11819 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:55:59.816210 11959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:55:59.816502 11959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:55:59.817648 11959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:55:59.819051 11959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:55:59.819157 11959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.819300 11959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:55:59.819376 11959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:55:59.854526 11959 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45979
I20260928 02:55:59.854580 12086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45979 every 8 connection(s)
I20260928 02:55:59.855858 11959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:55:59.863508 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 11959
I20260928 02:55:59.863726 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:55:59.864179 28567 external_mini_cluster.cc:1428] Setting key cb7bb3072bb883243dc7ec328f921238
Sep 28 02:55:59 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564159, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:55:59.876943 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.858436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:44515 (local address 127.27.229.254:35261)
0928 02:55:59.858655 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.858660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.859820 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 02:55:59.861691 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.861697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.861805 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.861918 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:59.863640 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.864552 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.866449 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.867106 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.867235 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.871069 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:59.871074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:59.871078 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:59.871113 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:59.873778 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.874513 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.874517 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.874519 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.874636 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.875149 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.875152 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.875154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.875422 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:59.875609 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.876467 (+ 858us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.876702 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:55:59.878198 12089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.858959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:44515)
0928 02:55:59.859668 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:59.859705 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:59.861412 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:59.862150 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:59.862164 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:59.862662 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:55:59.863398 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.863423 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.864715 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.864718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.866318 (+ 1600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.866331 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.867378 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.867387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.867740 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.868501 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:55:59.868533 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:55:59.870799 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:55:59.873936 (+ 3137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:59.873943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:59.873956 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:59.874394 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:59.874765 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:55:59.874769 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:55:59.874771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:55:59.875032 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:55:59.875604 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:55:59.875611 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:55:59.875980 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:55:59.876344 (+ 364us) client_negotiation.cc:240] Negotiation successful
0928 02:55:59.876861 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":280,"threads_started":1}
I20260928 02:55:59.880023 12087 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:55:59.880388 12087 heartbeater.cc:464] Registering TS with master...
I20260928 02:55:59.881191 12087 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
I20260928 02:55:59.882738 11630 ts_manager.cc:194] Registered new tserver with Master: b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:55:59.883597 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:44515
I20260928 02:55:59.892673 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:55:59.910712 11955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.897223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37934 (local address 127.27.229.254:35261)
0928 02:55:59.897480 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.897487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.897744 (+ 257us) server_negotiation.cc:407] Connection header received
0928 02:55:59.897959 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.897962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.898033 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.898172 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:55:59.899721 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.900501 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.901882 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.902313 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.902451 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.905257 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:55:59.905262 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:55:59.905265 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:55:59.905299 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:55:59.907985 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.908668 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.908675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.908679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.908812 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:55:59.909174 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:55:59.909179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:55:59.909183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:55:59.909501 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:55:59.909791 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.910147 (+ 356us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.910373 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:55:59.915011 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37934:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:55:59.921792 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:55:59.961740 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.934212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:53750)
0928 02:55:59.941146 (+ 6934us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:59.941204 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:59.941455 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:59.952330 (+ 10875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:59.952338 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:59.952382 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:59.952862 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.952885 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.955517 (+ 2632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.955526 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.958563 (+ 3037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.958578 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.960684 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.960691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.960900 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.960959 (+ 59us) client_negotiation.cc:769] Sending connection context
0928 02:55:59.961114 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 02:55:59.961323 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6579,"thread_start_us":362,"threads_started":1}
I20260928 02:55:59.963104 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.934432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.27.229.193:36789)
0928 02:55:59.951352 (+ 16920us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.951367 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.951419 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:55:59.951575 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.951588 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.951987 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.952295 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:59.953134 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.955290 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.958796 (+ 3506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.960475 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.960693 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.960883 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.961160 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.961333 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16619,"thread_start_us":18083,"threads_started":1}
I20260928 02:55:59.967130 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.935006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.0.0.1:46600)
0928 02:55:59.957063 (+ 22057us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:59.957092 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:59.957295 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:59.959650 (+ 2355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:59.959655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:59.959685 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:59.960085 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.960096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.962229 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.962234 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.964480 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.964490 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.966120 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.966127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.966362 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.966406 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 02:55:59.966499 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 02:55:59.966661 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21852,"mutex_wait_us":67,"thread_start_us":22595,"threads_started":1}
I20260928 02:55:59.967252 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.935412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:42687 (local address 127.0.0.1:60028)
0928 02:55:59.957938 (+ 22526us) negotiation.cc:107] Waiting for socket to connect
0928 02:55:59.957960 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:55:59.958099 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:55:59.959294 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:55:59.959298 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:55:59.959327 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:55:59.959696 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.959706 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.961593 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.961598 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.964188 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:55:59.964204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.966304 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.966310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:55:59.966557 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.966597 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 02:55:59.966686 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:55:59.966845 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22372,"mutex_wait_us":56,"thread_start_us":22513,"threads_started":1}
I20260928 02:55:59.967697 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.935286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46600 (local address 127.27.229.195:45979)
0928 02:55:59.958787 (+ 23501us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.958796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.958855 (+ 59us) server_negotiation.cc:407] Connection header received
0928 02:55:59.958960 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.958967 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.959176 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.959381 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:59.960275 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.961814 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.964651 (+ 2837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.965570 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.965783 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.965965 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.966973 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.967182 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23313,"thread_start_us":23155,"threads_started":1}
I20260928 02:55:59.967988 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:55:59.935959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.27.229.194:42687)
0928 02:55:59.956225 (+ 20266us) server_negotiation.cc:206] Beginning negotiation
0928 02:55:59.956238 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:55:59.958293 (+ 2055us) server_negotiation.cc:407] Connection header received
0928 02:55:59.958434 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:55:59.958444 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:55:59.958746 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:55:59.958976 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:55:59.959898 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.961432 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.964372 (+ 2940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:55:59.965788 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:55:59.965972 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:55:59.966127 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:55:59.967499 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0928 02:55:59.967679 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20014,"thread_start_us":20278,"threads_started":1}
I20260928 02:55:59.967588 11754 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc (DEFAULT_TABLE table=test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:59.969192 11754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8503e7ce0b4791a4edc5942039a4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:59.971269 11888 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc (DEFAULT_TABLE table=test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:59.971431 12022 tablet_service.cc:1467] Processing CreateTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc (DEFAULT_TABLE table=test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:55:59.972574 11888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8503e7ce0b4791a4edc5942039a4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:59.972776 12022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d8503e7ce0b4791a4edc5942039a4dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:55:59.984139 12105 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Bootstrap starting.
I20260928 02:55:59.986225 12106 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Bootstrap starting.
I20260928 02:55:59.987597 12107 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Bootstrap starting.
I20260928 02:55:59.988441 12105 tablet_bootstrap.cc:654] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:59.989813 12106 tablet_bootstrap.cc:654] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:59.989794 12105 log.cc:824] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:59.990593 12107 tablet_bootstrap.cc:654] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:55:59.991150 12106 log.cc:824] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:59.991874 12107 log.cc:824] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:55:59.993697 12105 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: No bootstrap required, opened a new log
I20260928 02:55:59.993986 12105 ts_tablet_manager.cc:1397] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260928 02:55:59.994585 12106 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: No bootstrap required, opened a new log
I20260928 02:55:59.994781 12107 tablet_bootstrap.cc:492] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: No bootstrap required, opened a new log
I20260928 02:55:59.994925 12106 ts_tablet_manager.cc:1397] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.004s
I20260928 02:55:59.995044 12107 ts_tablet_manager.cc:1397] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260928 02:56:00.003258 12106 raft_consensus.cc:348] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.003679 12106 raft_consensus.cc:374] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:00.003775 12106 raft_consensus.cc:729] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c69d4aa50b24edf82d8dc7046593ea6, State: Initialized, Role: FOLLOWER
I20260928 02:56:00.004438 12106 consensus_queue.cc:260] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.005823 12107 raft_consensus.cc:348] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.006304 12107 raft_consensus.cc:374] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:00.006120 12106 ts_tablet_manager.cc:1428] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260928 02:56:00.006402 12107 raft_consensus.cc:729] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dd8f580e4b4d98a2fe4b97c12b8d1a, State: Initialized, Role: FOLLOWER
I20260928 02:56:00.007781 12107 consensus_queue.cc:260] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.007717 12105 raft_consensus.cc:348] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.008203 12105 raft_consensus.cc:374] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:00.008319 12105 raft_consensus.cc:729] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ec59257b9449a86ec5ce128f248c0, State: Initialized, Role: FOLLOWER
I20260928 02:56:00.009718 12105 consensus_queue.cc:260] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.011525 12087 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:56:00.011826 12107 ts_tablet_manager.cc:1428] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Time spent starting tablet: real 0.017s user 0.012s sys 0.002s
I20260928 02:56:00.013710 11953 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:56:00.014268 12105 ts_tablet_manager.cc:1428] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Time spent starting tablet: real 0.020s user 0.013s sys 0.003s
W20260928 02:56:00.034276 11820 tablet.cc:2369] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:56:00.087044 11954 tablet.cc:2369] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:56:00.107818 12088 tablet.cc:2369] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:56:00.160948 12113 raft_consensus.cc:482] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:00.161347 12113 raft_consensus.cc:504] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.163506 12113 leader_election.cc:288] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:00.171007 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.164176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:59459 (local address 127.27.229.193:36789)
0928 02:56:00.164457 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.164466 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.165404 (+ 938us) server_negotiation.cc:407] Connection header received
0928 02:56:00.165614 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.165619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.165692 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.165795 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:00.166418 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.167208 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.169146 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.169989 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.170183 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.170266 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.170578 (+ 312us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.170705 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 02:56:00.171018 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.164209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37697 (local address 127.27.229.195:45979)
0928 02:56:00.164458 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.164467 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.165284 (+ 817us) server_negotiation.cc:407] Connection header received
0928 02:56:00.165524 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.165529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.165601 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.165703 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:00.166315 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.167141 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.169203 (+ 2062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.170019 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.170204 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.170284 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.170590 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.170716 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:56:00.171175 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.163852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.27.229.194:59459)
0928 02:56:00.165281 (+ 1429us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:00.165296 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:00.165456 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:00.165815 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:00.165819 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:00.165857 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:00.166263 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:00.166272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.167349 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.167354 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:00.168970 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:00.168982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.170228 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.170235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:00.170408 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.170435 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 02:56:00.170543 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 02:56:00.170679 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":3,"thread_start_us":1163,"threads_started":1}
I20260928 02:56:00.171650 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.163820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.27.229.194:37697)
0928 02:56:00.165161 (+ 1341us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:00.165184 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:00.165362 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:00.165719 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:00.165723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:00.165756 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:00.166149 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:00.166158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.167270 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.167277 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:00.169050 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:00.169070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.170258 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.170265 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:00.170422 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.170451 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 02:56:00.170560 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:56:00.170691 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"mutex_wait_us":8,"thread_start_us":913,"threads_started":1}
I20260928 02:56:00.172282 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8503e7ce0b4791a4edc5942039a4dc" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" is_pre_election: true
I20260928 02:56:00.172621 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8503e7ce0b4791a4edc5942039a4dc" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" is_pre_election: true
I20260928 02:56:00.172925 11774 raft_consensus.cc:2454] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 0.
I20260928 02:56:00.173111 12042 raft_consensus.cc:2454] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 0.
I20260928 02:56:00.173668 11841 leader_election.cc:302] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6, 324ec59257b9449a86ec5ce128f248c0; no voters:
I20260928 02:56:00.174043 12113 raft_consensus.cc:2781] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:56:00.174201 12113 raft_consensus.cc:482] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:56:00.174314 12113 raft_consensus.cc:3037] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:00.176684 12113 raft_consensus.cc:504] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.177304 12113 leader_election.cc:288] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 election: Requested vote from peers 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:00.177728 11774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8503e7ce0b4791a4edc5942039a4dc" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
I20260928 02:56:00.177812 12042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d8503e7ce0b4791a4edc5942039a4dc" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
I20260928 02:56:00.177937 11774 raft_consensus.cc:3037] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:00.178047 12042 raft_consensus.cc:3037] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:00.180490 11774 raft_consensus.cc:2454] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 1.
I20260928 02:56:00.180568 12042 raft_consensus.cc:2454] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 1.
I20260928 02:56:00.180958 11841 leader_election.cc:302] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6, 324ec59257b9449a86ec5ce128f248c0; no voters:
I20260928 02:56:00.181300 12113 raft_consensus.cc:2781] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:56:00.181721 12113 raft_consensus.cc:686] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 1 LEADER]: Becoming Leader. State: Replica: 324ec59257b9449a86ec5ce128f248c0, State: Running, Role: LEADER
I20260928 02:56:00.182272 12113 consensus_queue.cc:237] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:00.186957 11629 catalog_manager.cc:5495] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "324ec59257b9449a86ec5ce128f248c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260928 02:56:00.211493 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.205642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.27.229.194:42687)
0928 02:56:00.205830 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.205835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.205852 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:56:00.206080 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.206085 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.206173 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.206303 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:56:00.206729 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.207608 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.209053 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.209625 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.209875 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.210360 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:56:00.210941 (+ 581us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:56:00.211052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.211170 (+ 118us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.211264 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:56:00.219666 12042 raft_consensus.cc:1261] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Refusing update from remote peer 324ec59257b9449a86ec5ce128f248c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:56:00.219669 11774 raft_consensus.cc:1261] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Refusing update from remote peer 324ec59257b9449a86ec5ce128f248c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:56:00.220638 12116 consensus_queue.cc:1035] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:56:00.220978 12113 consensus_queue.cc:1035] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:56:00.232980 12121 mvcc.cc:205] Tried to move back new op lower bound from 7334150800249823232 to 7334150800110804992. Current Snapshot: MvccSnapshot[applied={T|T < 7334150800249823232}]
I20260928 02:56:00.233330 12119 mvcc.cc:205] Tried to move back new op lower bound from 7334150800249823232 to 7334150800110804992. Current Snapshot: MvccSnapshot[applied={T|T < 7334150800249823232}]
I20260928 02:56:00.245180 11629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:56:00.245508 11629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37934:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:56:00.249466 11629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Sending DeleteTablet for 3 replicas of tablet 7d8503e7ce0b4791a4edc5942039a4dc
I20260928 02:56:00.250970 12022 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:00 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:46600
I20260928 02:56:00.250993 11888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:00 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:60028
I20260928 02:56:00.250986 11754 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d8503e7ce0b4791a4edc5942039a4dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:00 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:53750
I20260928 02:56:00.253062 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11691
I20260928 02:56:00.281860 12128 tablet_replica.cc:330] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: stopping tablet replica
I20260928 02:56:00.283273 12128 raft_consensus.cc:2229] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:56:00.287833 12128 raft_consensus.cc:2258] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:00.293202 12128 ts_tablet_manager.cc:1905] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:56:00.296764 12129 tablet_replica.cc:330] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: stopping tablet replica
I20260928 02:56:00.297197 12129 raft_consensus.cc:2229] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:56:00.297673 12129 raft_consensus.cc:2258] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:00.298355 12130 tablet_replica.cc:330] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: stopping tablet replica
I20260928 02:56:00.298712 12130 raft_consensus.cc:2229] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:56:00.299154 12130 raft_consensus.cc:2258] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:00.301324 12129 ts_tablet_manager.cc:1905] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:56:00.301867 12130 ts_tablet_manager.cc:1905] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:56:00 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:56:00.308738 12129 ts_tablet_manager.cc:1918] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:56:00.308843 12128 ts_tablet_manager.cc:1918] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:56:00.309026 12128 log.cc:1196] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/7d8503e7ce0b4791a4edc5942039a4dc
I20260928 02:56:00.309051 12129 log.cc:1196] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/7d8503e7ce0b4791a4edc5942039a4dc
I20260928 02:56:00.309602 12128 ts_tablet_manager.cc:1939] T 7d8503e7ce0b4791a4edc5942039a4dc P 324ec59257b9449a86ec5ce128f248c0: Deleting consensus metadata
I20260928 02:56:00.309932 12129 ts_tablet_manager.cc:1939] T 7d8503e7ce0b4791a4edc5942039a4dc P 2c69d4aa50b24edf82d8dc7046593ea6: Deleting consensus metadata
I20260928 02:56:00.310784 12130 ts_tablet_manager.cc:1918] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:56:00.311004 12130 log.cc:1196] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/7d8503e7ce0b4791a4edc5942039a4dc
I20260928 02:56:00.311734 12130 ts_tablet_manager.cc:1939] T 7d8503e7ce0b4791a4edc5942039a4dc P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting consensus metadata
I20260928 02:56:00.311664 11614 catalog_manager.cc:4841] TS 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): tablet 7d8503e7ce0b4791a4edc5942039a4dc (table test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]) successfully deleted
I20260928 02:56:00.311997 11614 catalog_manager.cc:4841] TS 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789): tablet 7d8503e7ce0b4791a4edc5942039a4dc (table test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]) successfully deleted
I20260928 02:56:00.313823 11615 catalog_manager.cc:4841] TS b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): tablet 7d8503e7ce0b4791a4edc5942039a4dc (table test-table [id=9dd545f3c9ff4582801e7cbfe9fc91d4]) successfully deleted
I20260928 02:56:00.319020 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.254921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53766 (local address 127.27.229.193:36789)
0928 02:56:00.255220 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.255229 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.277535 (+ 22306us) server_negotiation.cc:407] Connection header received
0928 02:56:00.277910 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.277920 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.278052 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.278147 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:00.283739 (+ 5592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.285729 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.293578 (+ 7849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.294636 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.294794 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.312351 (+ 17557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:00.312364 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:00.312382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:00.312426 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:00.315956 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.316832 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.316839 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.316842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.316973 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.317391 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.317396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.317398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.317748 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:00.318064 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.318406 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.318629 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260928 02:56:00.319715 11814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:00.427443 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11691
I20260928 02:56:00.445966 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11825
Sep 28 02:56:00 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:56:00.463295 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.446789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60042 (local address 127.27.229.194:42687)
0928 02:56:00.447035 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.447040 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.447057 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:56:00.447232 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.447236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.447298 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.447387 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:00.449019 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.449995 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.452066 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.452538 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.452694 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.457060 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:00.457071 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:00.457086 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:00.457122 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:00.460342 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.461050 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.461057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.461059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.461199 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.461600 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.461604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.461606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.461974 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:00.462311 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.462614 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.462911 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141}
I20260928 02:56:00.464064 11948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:00.582721 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11825
I20260928 02:56:00.602231 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11959
Sep 28 02:56:00 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564157, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:56:00.617569 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:00.603087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46616 (local address 127.27.229.195:45979)
0928 02:56:00.603316 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:00.603321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:00.603341 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:56:00.603449 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:00.603453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:00.603514 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:00.603614 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:00.604915 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.605650 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.607123 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:00.607568 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:00.607678 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:00.612144 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:00.612159 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:00.612175 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:00.612218 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:00.615054 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.615717 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.615724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.615727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.615883 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:00.616262 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:00.616265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:00.616267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:00.616592 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:00.616807 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:00.617105 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 02:56:00.617307 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:56:00.618305 12082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:00.709587 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11959
I20260928 02:56:00.726539 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 11598
I20260928 02:56:00.727792 11660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:00.840999 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 11598
I20260928 02:56:20.864313 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--webserver_interface=127.27.229.254
--webserver_port=42795
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:56:21.206575 12136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:21.206972 12136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:21.207144 12136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:21.214648 12136 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:56:21.214787 12136 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:56:21.214895 12136 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:56:21.214959 12136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:56:21.215027 12136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:21.215154 12136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:21.215220 12136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:56:21.215297 12136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:56:21.225379 12136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=42795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:21.226698 12136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:21.228142 12136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:21.252511 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:21.252417 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:21.252467 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:21.254372 12136 server_base.cc:1034] running on GCE node
I20260928 02:56:21.255399 12136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:21.257010 12136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:21.258245 12136 hybrid_clock.cc:648] HybridClock initialized: now 1790564181258162 us; error 56 us; skew 500 ppm
I20260928 02:56:21.258668 12136 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:21 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:21.262440 12136 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:56:21.264346 12136 webserver.cc:460] Webserver started at http://127.27.229.254:42795/ using document root <none> and password file <none>
I20260928 02:56:21.265137 12136 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:21.265249 12136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:21.271608 12136 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 02:56:21.275067 12152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:21.276573 12136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:56:21.276724 12136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "c9de88825dbf4554b6ae456623a68c20"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "799327b22bb4ef338fc4af846bde8253"
server_key_iv: "3282bb88e8d1a2cf5959d49191a6326b"
server_key_version: "encryptionkey@0"
I20260928 02:56:21.277797 12136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:21.294519 12136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:21.298997 12136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:21.299315 12136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:21.319416 12136 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:35261
I20260928 02:56:21.319448 12203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:35261 every 8 connection(s)
I20260928 02:56:21.321070 12136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:56:21.328316 12204 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:56:21.329388 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12136
I20260928 02:56:21.330406 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:36789
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=42459
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:56:21.333542 12204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap starting.
I20260928 02:56:21.355945 12204 log.cc:824] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:21.371448 12204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:56:21.372030 12204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap complete.
I20260928 02:56:21.381752 12204 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:21.382709 12204 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Initialized, Role: FOLLOWER
I20260928 02:56:21.383414 12204 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:21.383666 12204 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:56:21.383795 12204 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:56:21.383983 12204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 1 FOLLOWER]: Advancing to term 2
I20260928 02:56:21.388016 12204 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:21.388655 12204 leader_election.cc:302] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9de88825dbf4554b6ae456623a68c20; no voters:
I20260928 02:56:21.389348 12204 leader_election.cc:288] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 02:56:21.389493 12208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 02:56:21.390945 12208 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 LEADER]: Becoming Leader. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Running, Role: LEADER
I20260928 02:56:21.391629 12208 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:21.391911 12204 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:56:21.394233 12210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9de88825dbf4554b6ae456623a68c20. Latest consensus state: current_term: 2 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:56:21.394624 12210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:56:21.394752 12209 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:56:21.395205 12209 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:56:21.402570 12216 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:56:21.411087 12216 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:56:21.413044 12216 catalog_manager.cc:1259] Loaded cluster ID: 58f040dd68014adc8713415808340d6f
I20260928 02:56:21.413239 12216 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:56:21.417804 12225 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:56:21.418395 12216 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:56:21.419915 12216 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Loaded TSK: 0
I20260928 02:56:21.447768 12216 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Generated new TSK 1
I20260928 02:56:21.448215 12216 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:56:21.716629 12206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:21.716910 12206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:21.717063 12206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:21.724517 12206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:21.724639 12206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:21.724838 12206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:56:21.735162 12206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:36789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=42459
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:21.736493 12206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:21.737831 12206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:21.753718 12231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:21.753729 12232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:21.754268 12206 server_base.cc:1034] running on GCE node
W20260928 02:56:21.755064 12234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:21.755776 12206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:21.756830 12206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:21.758143 12206 hybrid_clock.cc:648] HybridClock initialized: now 1790564181758084 us; error 50 us; skew 500 ppm
I20260928 02:56:21.758495 12206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:21 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:21.761888 12206 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:56:21.763587 12206 webserver.cc:460] Webserver started at http://127.27.229.193:42459/ using document root <none> and password file <none>
I20260928 02:56:21.764333 12206 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:21.764482 12206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:21.770280 12206 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 02:56:21.773382 12241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:21.775157 12206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:56:21.775362 12206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "a18d53795922bcd0ab0b182d42c06ea7"
server_key_iv: "755b2fdfd755041770604bbd3fbc8d35"
server_key_version: "encryptionkey@0"
I20260928 02:56:21.776626 12206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:21.789455 12206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:21.794494 12206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:21.794782 12206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:21.796010 12206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:21.797400 12206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:56:21.797529 12206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:21.797681 12206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:56:21.797806 12206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260928 02:56:21.833375 12206 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:36789
I20260928 02:56:21.833408 12353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:36789 every 8 connection(s)
I20260928 02:56:21.835467 12206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:56:21.839042 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12206
I20260928 02:56:21.839815 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:42687
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=36233
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:56:21 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564181, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:56:21.871686 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:21.839417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:39159)
0928 02:56:21.840397 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:21.840459 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:21.843411 (+ 2952us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:21.849337 (+ 5926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:21.849355 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:21.850256 (+ 901us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:56:21.851169 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:21.851191 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:21.853787 (+ 2596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:21.853793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:21.856097 (+ 2304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:21.856115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:21.857514 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:21.857522 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:21.857849 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:21.858640 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:56:21.858665 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:56:21.861377 (+ 2712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:56:21.866819 (+ 5442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:21.866884 (+ 65us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:21.866909 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:21.867441 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:21.868111 (+ 670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:21.868115 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:21.868117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:21.868316 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:21.869488 (+ 1172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:56:21.869494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:56:21.869804 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 02:56:21.870037 (+ 233us) client_negotiation.cc:240] Negotiation successful
0928 02:56:21.870506 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":1470,"threads_started":1}
I20260928 02:56:21.871928 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:21.840825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:39159 (local address 127.27.229.254:35261)
0928 02:56:21.847426 (+ 6601us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:21.847447 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:21.847505 (+ 58us) server_negotiation.cc:407] Connection header received
0928 02:56:21.847992 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:21.848020 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:21.848586 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:21.851427 (+ 2841us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:21.851919 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:21.853502 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:21.856302 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:21.857214 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:21.857472 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:21.861852 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:21.861870 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:21.861896 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:21.861964 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:21.866603 (+ 4639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:21.867626 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:21.867639 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:21.867648 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:21.867906 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:21.868467 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:21.868475 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:21.868481 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:21.869307 (+ 826us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:21.869545 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:21.870313 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 02:56:21.870748 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6311,"thread_start_us":6148,"threads_started":1}
I20260928 02:56:21.874809 12354 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:56:21.875191 12354 heartbeater.cc:464] Registering TS with master...
I20260928 02:56:21.875922 12354 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
I20260928 02:56:21.878284 12169 ts_manager.cc:194] Registered new tserver with Master: 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789)
I20260928 02:56:21.879896 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:39159
W20260928 02:56:22.191462 12358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:22.191802 12358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:22.191972 12358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:22.199631 12358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:22.199790 12358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:22.199976 12358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:56:22.210124 12358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:42687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=36233
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:22.211519 12358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:22.212819 12358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:22.226140 12366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:22.226356 12364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:22.226861 12363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:22.226974 12358 server_base.cc:1034] running on GCE node
I20260928 02:56:22.227620 12358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:22.228571 12358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:22.229809 12358 hybrid_clock.cc:648] HybridClock initialized: now 1790564182229734 us; error 69 us; skew 500 ppm
I20260928 02:56:22.230134 12358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:22 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:22.233335 12358 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:56:22.234793 12358 webserver.cc:460] Webserver started at http://127.27.229.194:36233/ using document root <none> and password file <none>
I20260928 02:56:22.235513 12358 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:22.235625 12358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:22.240882 12358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 02:56:22.243844 12373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.245333 12358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:56:22.245482 12358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "324ec59257b9449a86ec5ce128f248c0"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "d0366868a69d78819d04f15e6555bfcd"
server_key_iv: "3478b1b64c48c933ff5aac35b86a1c64"
server_key_version: "encryptionkey@0"
I20260928 02:56:22.246430 12358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:22.260665 12358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:22.264998 12358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:22.265231 12358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:22.266194 12358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:22.267645 12358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:56:22.267784 12358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.267910 12358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:56:22.267997 12358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.303401 12358 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:42687
I20260928 02:56:22.303439 12485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:42687 every 8 connection(s)
I20260928 02:56:22.304900 12358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:56:22.310519 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12358
I20260928 02:56:22.311291 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:45979
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=46311
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 02:56:22 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:56:22.326642 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.307490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:39289 (local address 127.27.229.254:35261)
0928 02:56:22.307669 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.307674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.308799 (+ 1125us) server_negotiation.cc:407] Connection header received
0928 02:56:22.310610 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.310618 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.310715 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.310857 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:22.312482 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.313658 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.315431 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.316229 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.316375 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.320522 (+ 4147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:22.320528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:22.320531 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:22.320566 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:22.323392 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.324206 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.324211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.324213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.324342 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.324804 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.324808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.324810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.325135 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:22.325333 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.326034 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.326311 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:56:22.327689 12488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.307904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:39289)
0928 02:56:22.308602 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:22.308641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:22.310320 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:22.311089 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:22.311098 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:22.311724 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:56:22.312297 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.312315 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.313865 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.313874 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.315272 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.315285 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.316460 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.316473 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.316681 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.317432 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:56:22.317456 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:56:22.320251 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:56:22.323557 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:22.323566 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:22.323582 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:22.324040 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:22.324461 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:22.324466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:22.324470 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:22.324665 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:22.325290 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:56:22.325300 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:56:22.325657 (+ 357us) client_negotiation.cc:769] Sending connection context
0928 02:56:22.325936 (+ 279us) client_negotiation.cc:240] Negotiation successful
0928 02:56:22.326487 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":279,"threads_started":1}
I20260928 02:56:22.329411 12486 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:56:22.329798 12486 heartbeater.cc:464] Registering TS with master...
I20260928 02:56:22.330550 12486 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
I20260928 02:56:22.332154 12169 ts_manager.cc:194] Registered new tserver with Master: 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687)
I20260928 02:56:22.333060 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:39289
W20260928 02:56:22.658005 12489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:22.658319 12489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:22.658463 12489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:22.665968 12489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:22.666131 12489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:22.666283 12489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:56:22.676339 12489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:45979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=46311
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:22.677623 12489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:22.678891 12489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:22.693773 12495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:22.693755 12497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:22.693833 12494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:22.694742 12489 server_base.cc:1034] running on GCE node
I20260928 02:56:22.695487 12489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:22.696600 12489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:22.697824 12489 hybrid_clock.cc:648] HybridClock initialized: now 1790564182697773 us; error 41 us; skew 500 ppm
I20260928 02:56:22.698167 12489 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:22 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:22.701859 12489 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:56:22.703727 12489 webserver.cc:460] Webserver started at http://127.27.229.195:46311/ using document root <none> and password file <none>
I20260928 02:56:22.704512 12489 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:22.704682 12489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:22.711030 12489 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:56:22.714123 12504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.715896 12489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:56:22.716125 12489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "e151992d0192a90e17edc618a5b83812"
server_key_iv: "650f7fedb5020b5a3a75e112a6eaebc6"
server_key_version: "encryptionkey@0"
I20260928 02:56:22.717240 12489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:22.737316 12489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:22.741755 12489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:22.742057 12489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:22.743175 12489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:22.744581 12489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:56:22.744680 12489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.744752 12489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:56:22.744820 12489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:22.781730 12489 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45979
I20260928 02:56:22.781804 12616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45979 every 8 connection(s)
I20260928 02:56:22.783516 12489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:56:22.792824 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12489
Sep 28 02:56:22 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564182, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:56:22.809701 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.786436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:56177 (local address 127.27.229.254:35261)
0928 02:56:22.786667 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.786672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.788032 (+ 1360us) server_negotiation.cc:407] Connection header received
0928 02:56:22.790276 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.790283 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.790383 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.790498 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:22.792298 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.793491 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.796113 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.796838 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.797002 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.801248 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:22.801257 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:22.801263 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:22.801307 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:22.805415 (+ 4108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.806557 (+ 1142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.806565 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.806569 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.806734 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.807398 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.807407 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.807413 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.807823 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:22.807996 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.809012 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.809289 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 02:56:22.810751 12619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.787058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:56177)
0928 02:56:22.787861 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:22.787901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:22.789944 (+ 2043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:22.790761 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:22.790770 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:22.791386 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:56:22.792033 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.792052 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.793859 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.793868 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.795957 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.795971 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.797100 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.797105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.797357 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.798243 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:56:22.798278 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:56:22.800988 (+ 2710us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:56:22.805589 (+ 4601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:22.805602 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:22.805632 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:22.806341 (+ 709us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:22.806930 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:22.806938 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:22.806944 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:22.807224 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:22.807984 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:56:22.807997 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:56:22.808497 (+ 500us) client_negotiation.cc:769] Sending connection context
0928 02:56:22.808870 (+ 373us) client_negotiation.cc:240] Negotiation successful
0928 02:56:22.809403 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
I20260928 02:56:22.812386 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.794241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52096 (local address 127.27.229.254:35261)
0928 02:56:22.794750 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.794758 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.795146 (+ 388us) server_negotiation.cc:407] Connection header received
0928 02:56:22.795379 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.795386 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.795465 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.795573 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:22.797632 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.798733 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.800261 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.800927 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.801064 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.805421 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:22.805427 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:22.805430 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:22.805465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:22.809049 (+ 3584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.810029 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.810033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.810035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.810143 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.810623 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.810627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.810629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.811097 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:22.811466 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.811777 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.812015 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":246,"threads_started":1}
I20260928 02:56:22.812600 12617 heartbeater.cc:347] Connected to a master server at 127.27.229.254:35261
I20260928 02:56:22.812911 12617 heartbeater.cc:464] Registering TS with master...
I20260928 02:56:22.813655 12617 heartbeater.cc:511] Master 127.27.229.254:35261 requested a full tablet report, sending...
I20260928 02:56:22.815088 12169 ts_manager.cc:194] Registered new tserver with Master: b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:22.815938 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:56177
I20260928 02:56:22.827623 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:56:22.834929 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.828647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.27.229.254:35261)
0928 02:56:22.828889 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.828899 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.829256 (+ 357us) server_negotiation.cc:407] Connection header received
0928 02:56:22.829505 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.829510 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.829661 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.829809 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:56:22.830252 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.831070 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.832761 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.833206 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.833358 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.833938 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:56:22.834191 (+ 253us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 02:56:22.834609 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52100: token expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:56:22.835085 12620 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52100: token expired
W20260928 02:56:22.835669 12091 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.229.254:35261)
I20260928 02:56:22.850087 12620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.836245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52104 (local address 127.27.229.254:35261)
0928 02:56:22.836399 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.836403 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.836492 (+ 89us) server_negotiation.cc:407] Connection header received
0928 02:56:22.836642 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.836646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.836706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.836781 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:22.838573 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.839413 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.840801 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.841240 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.841372 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.844166 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:22.844171 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:22.844175 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:22.844210 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:22.846988 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.847884 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.847889 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.847891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.848030 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:22.848431 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:22.848438 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:22.848444 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:22.848875 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:22.849065 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.849481 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.849726 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 02:56:22.853783 12169 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52104:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:56:22.862020 12169 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:56:22.883484 12354 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:56:22.896337 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.875397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.0.0.1:56448)
0928 02:56:22.886018 (+ 10621us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:22.886052 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:22.886256 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:22.887915 (+ 1659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:22.887922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:22.887980 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:22.888610 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.888624 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890761 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.890766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.893142 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.893156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.895145 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.895362 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.895420 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 02:56:22.895622 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 02:56:22.895841 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10427,"mutex_wait_us":226,"spinlock_wait_cycles":1408,"thread_start_us":11605,"threads_started":1}
I20260928 02:56:22.896817 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.876764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33312 (local address 127.27.229.194:42687)
0928 02:56:22.883885 (+ 7121us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.883898 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.887347 (+ 3449us) server_negotiation.cc:407] Connection header received
0928 02:56:22.887656 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.887663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.887877 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.888036 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:22.889007 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890511 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.894292 (+ 3781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.895638 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895791 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.895929 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.896256 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.896453 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6853,"thread_start_us":6705,"threads_started":1}
I20260928 02:56:22.896823 12627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.874711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47300)
0928 02:56:22.875591 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:22.875651 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:22.875896 (+ 245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:22.887912 (+ 12016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:22.887921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:22.887967 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:22.888532 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.888546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890578 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.890586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.893459 (+ 2873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.893473 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.895594 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.895767 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.895814 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 02:56:22.895912 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:56:22.896057 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":208,"spinlock_wait_cycles":3840,"thread_start_us":364,"threads_started":1}
I20260928 02:56:22.896961 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.874980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47300 (local address 127.27.229.193:36789)
0928 02:56:22.887244 (+ 12264us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.887253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.887281 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:56:22.887372 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.887383 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.887665 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.888553 (+ 888us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:22.888779 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890410 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.893627 (+ 3217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.895360 (+ 1733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895586 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.895742 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.896457 (+ 715us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.896625 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12097,"thread_start_us":11597,"threads_started":1}
I20260928 02:56:22.897428 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.877332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.27.229.195:45979)
0928 02:56:22.885630 (+ 8298us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:22.885639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:22.886375 (+ 736us) server_negotiation.cc:407] Connection header received
0928 02:56:22.887389 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:22.887399 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:22.887664 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:22.888826 (+ 1162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:22.889159 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890613 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.893358 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.894923 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895122 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.895305 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:22.896867 (+ 1562us) server_negotiation.cc:299] Negotiation successful
0928 02:56:22.897082 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8070,"thread_start_us":7863,"threads_started":1}
I20260928 02:56:22.897927 12629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:22.875931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:42687 (local address 127.0.0.1:33312)
0928 02:56:22.887227 (+ 11296us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:22.887246 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:22.887463 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:22.888343 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:22.888347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:22.888376 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:22.888835 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.888847 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.890654 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.890662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.894133 (+ 3471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:22.894146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:22.895904 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:22.895910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:22.896068 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:22.896098 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 02:56:22.896194 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 02:56:22.896337 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11173,"mutex_wait_us":57,"thread_start_us":9696,"threads_started":1}
I20260928 02:56:22.900727 12552 tablet_service.cc:1467] Processing CreateTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f (DEFAULT_TABLE table=test-table [id=138761cdc64348ddbe381d95d41d9e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:56:22.901013 12289 tablet_service.cc:1467] Processing CreateTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f (DEFAULT_TABLE table=test-table [id=138761cdc64348ddbe381d95d41d9e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:56:22.901233 12421 tablet_service.cc:1467] Processing CreateTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f (DEFAULT_TABLE table=test-table [id=138761cdc64348ddbe381d95d41d9e43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:56:22.902132 12552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598e90479c0a4ad7803b153e1bf7e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:56:22.902441 12421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598e90479c0a4ad7803b153e1bf7e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:56:22.902447 12289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 598e90479c0a4ad7803b153e1bf7e97f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:56:22.914803 12633 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Bootstrap starting.
I20260928 02:56:22.918465 12633 tablet_bootstrap.cc:654] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Neither blocks nor log segments found. Creating new log.
I20260928 02:56:22.919626 12633 log.cc:824] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:22.922801 12633 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: No bootstrap required, opened a new log
I20260928 02:56:22.923163 12633 ts_tablet_manager.cc:1397] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260928 02:56:22.923601 12635 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Bootstrap starting.
I20260928 02:56:22.925793 12634 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Bootstrap starting.
I20260928 02:56:22.927033 12635 tablet_bootstrap.cc:654] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Neither blocks nor log segments found. Creating new log.
I20260928 02:56:22.928432 12635 log.cc:824] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:22.929458 12634 tablet_bootstrap.cc:654] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Neither blocks nor log segments found. Creating new log.
I20260928 02:56:22.930702 12634 log.cc:824] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:22.931809 12635 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: No bootstrap required, opened a new log
I20260928 02:56:22.931672 12633 raft_consensus.cc:348] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.932122 12635 ts_tablet_manager.cc:1397] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260928 02:56:22.932149 12633 raft_consensus.cc:374] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:22.932246 12633 raft_consensus.cc:729] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7dd8f580e4b4d98a2fe4b97c12b8d1a, State: Initialized, Role: FOLLOWER
I20260928 02:56:22.933015 12633 consensus_queue.cc:260] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.933933 12634 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: No bootstrap required, opened a new log
I20260928 02:56:22.934207 12634 ts_tablet_manager.cc:1397] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260928 02:56:22.934234 12617 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:56:22.934885 12633 ts_tablet_manager.cc:1428] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Time spent starting tablet: real 0.012s user 0.005s sys 0.007s
I20260928 02:56:22.941203 12635 raft_consensus.cc:348] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.941574 12635 raft_consensus.cc:374] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:22.941643 12635 raft_consensus.cc:729] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 324ec59257b9449a86ec5ce128f248c0, State: Initialized, Role: FOLLOWER
I20260928 02:56:22.942204 12635 consensus_queue.cc:260] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.943362 12486 heartbeater.cc:503] Master 127.27.229.254:35261 was elected leader, sending a full tablet report...
I20260928 02:56:22.943812 12635 ts_tablet_manager.cc:1428] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260928 02:56:22.943727 12634 raft_consensus.cc:348] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.944186 12634 raft_consensus.cc:374] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:56:22.944275 12634 raft_consensus.cc:729] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c69d4aa50b24edf82d8dc7046593ea6, State: Initialized, Role: FOLLOWER
I20260928 02:56:22.944895 12634 consensus_queue.cc:260] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:22.946645 12634 ts_tablet_manager.cc:1428] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
W20260928 02:56:23.035290 12618 tablet.cc:2369] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:56:23.056938 12487 tablet.cc:2369] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:56:23.089753 12355 tablet.cc:2369] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:56:23.122332 12640 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:23.122699 12640 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:23.124001 12641 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:23.124310 12641 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:23.125932 12640 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:23.127199 12641 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:23.135506 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.125369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:46743 (local address 127.27.229.193:36789)
0928 02:56:23.125632 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.125640 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.126098 (+ 458us) server_negotiation.cc:407] Connection header received
0928 02:56:23.126431 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.126437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.126523 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.126587 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:23.128784 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.129975 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.132809 (+ 2834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.134147 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.134284 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.134366 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.135050 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.135204 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":95}
I20260928 02:56:23.135946 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.127132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:43069 (local address 127.27.229.194:42687)
0928 02:56:23.127332 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.127338 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.127998 (+ 660us) server_negotiation.cc:407] Connection header received
0928 02:56:23.128305 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.128313 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.128390 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.128456 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:23.129430 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.130689 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.133502 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.134771 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.134899 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.134974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.135538 (+ 564us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.135691 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":60}
I20260928 02:56:23.136010 12632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.126415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33179 (local address 127.27.229.195:45979)
0928 02:56:23.126606 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.126611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.127235 (+ 624us) server_negotiation.cc:407] Connection header received
0928 02:56:23.127510 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.127517 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.127604 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.127667 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:23.128605 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.129813 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.133508 (+ 3695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.134782 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.134957 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.135030 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.135575 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.135721 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:56:23.136680 12644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.126626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:42687 (local address 127.27.229.193:43069)
0928 02:56:23.127885 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.127905 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:23.128106 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:23.128698 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:23.128703 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:23.128748 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:23.129260 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.129278 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.130917 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.130922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.133233 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.133247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.135154 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.135160 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.135335 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.135367 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:56:23.136118 (+ 751us) client_negotiation.cc:240] Negotiation successful
0928 02:56:23.136266 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1107,"mutex_wait_us":53,"thread_start_us":907,"threads_started":1}
I20260928 02:56:23.136696 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.125160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.27.229.194:46743)
0928 02:56:23.125984 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.126005 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:23.126253 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:23.126867 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:23.126873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:23.126909 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:23.128668 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.128680 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.130236 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.130247 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.132582 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.132598 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.134588 (+ 1990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.134595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.134862 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.134897 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:56:23.135908 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0928 02:56:23.136301 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"mutex_wait_us":36,"thread_start_us":405,"threads_started":1}
I20260928 02:56:23.137149 12645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.127405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.27.229.193:48971)
0928 02:56:23.128134 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.128152 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:23.128261 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:23.129629 (+ 1368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:23.129634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:23.129657 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:23.130068 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.130080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.131623 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.131632 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.134207 (+ 2575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.134223 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.136176 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.136184 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.136470 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.136508 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 02:56:23.136608 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 02:56:23.136744 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"mutex_wait_us":1,"thread_start_us":606,"threads_started":1}
I20260928 02:56:23.137411 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.127798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:48971 (local address 127.27.229.195:45979)
0928 02:56:23.129304 (+ 1506us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.129316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.129340 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:56:23.129402 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.129409 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.129482 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.130229 (+ 747us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:56:23.130474 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.131431 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.134387 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.135792 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.135935 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.136004 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.136903 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.137058 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":2278,"threads_started":1}
I20260928 02:56:23.137781 12441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324ec59257b9449a86ec5ce128f248c0" is_pre_election: true
I20260928 02:56:23.138116 12309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" is_pre_election: true
I20260928 02:56:23.138309 12441 raft_consensus.cc:2454] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c69d4aa50b24edf82d8dc7046593ea6 in term 0.
I20260928 02:56:23.138209 12572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" is_pre_election: true
I20260928 02:56:23.138309 12643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.126169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.27.229.194:33179)
0928 02:56:23.127123 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.127148 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:23.127344 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:23.127883 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:23.127890 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:23.127921 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:56:23.128351 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.128369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.129997 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.130005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.133311 (+ 3306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:23.133329 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.135206 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.135214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:23.135392 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.135426 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:56:23.136503 (+ 1077us) client_negotiation.cc:240] Negotiation successful
0928 02:56:23.136670 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":844,"thread_start_us":597,"threads_started":1}
I20260928 02:56:23.138571 12309 raft_consensus.cc:2454] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 0.
I20260928 02:56:23.138732 12572 raft_consensus.cc:2454] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c69d4aa50b24edf82d8dc7046593ea6 in term 0.
I20260928 02:56:23.139238 12374 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6, 324ec59257b9449a86ec5ce128f248c0; no voters:
I20260928 02:56:23.139237 12242 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6, 324ec59257b9449a86ec5ce128f248c0; no voters:
I20260928 02:56:23.139514 12572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" is_pre_election: true
I20260928 02:56:23.139664 12640 raft_consensus.cc:2781] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:56:23.139715 12572 raft_consensus.cc:2454] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 0.
I20260928 02:56:23.139814 12640 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:56:23.139849 12641 raft_consensus.cc:2781] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:56:23.139919 12640 raft_consensus.cc:3037] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:23.140015 12641 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:56:23.140137 12641 raft_consensus.cc:3037] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:23.142196 12640 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:23.142299 12641 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:23.142908 12640 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 election: Requested vote from peers 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:23.143313 12309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
I20260928 02:56:23.143339 12441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "324ec59257b9449a86ec5ce128f248c0"
I20260928 02:56:23.143356 12572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "324ec59257b9449a86ec5ce128f248c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
I20260928 02:56:23.143585 12309 raft_consensus.cc:2379] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 324ec59257b9449a86ec5ce128f248c0 in current term 1: Already voted for candidate 2c69d4aa50b24edf82d8dc7046593ea6 in this term.
I20260928 02:56:23.143540 12572 raft_consensus.cc:3037] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:56:23.143615 12441 raft_consensus.cc:2379] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2c69d4aa50b24edf82d8dc7046593ea6 in current term 1: Already voted for candidate 324ec59257b9449a86ec5ce128f248c0 in this term.
I20260928 02:56:23.144549 12641 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 1 election: Requested vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
I20260928 02:56:23.144726 12571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "598e90479c0a4ad7803b153e1bf7e97f" candidate_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
I20260928 02:56:23.145519 12243 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:23.145840 12641 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260928 02:56:23.146514 12572 raft_consensus.cc:2454] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 324ec59257b9449a86ec5ce128f248c0 in term 1.
I20260928 02:56:23.147010 12375 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a; no voters: 2c69d4aa50b24edf82d8dc7046593ea6
I20260928 02:56:23.147460 12640 raft_consensus.cc:2781] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:56:23.147956 12640 raft_consensus.cc:686] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 LEADER]: Becoming Leader. State: Replica: 324ec59257b9449a86ec5ce128f248c0, State: Running, Role: LEADER
I20260928 02:56:23.148486 12640 consensus_queue.cc:237] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:23.153086 12168 catalog_manager.cc:5495] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "324ec59257b9449a86ec5ce128f248c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260928 02:56:23.194669 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.189001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.27.229.194:42687)
0928 02:56:23.189188 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.189196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.189264 (+ 68us) server_negotiation.cc:407] Connection header received
0928 02:56:23.189494 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.189498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.189576 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.189729 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:56:23.190136 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.190943 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.192476 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.192907 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.193039 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.193527 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:56:23.194037 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:56:23.194211 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.194302 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.194385 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:56:23.203575 12572 raft_consensus.cc:1261] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Refusing update from remote peer 324ec59257b9449a86ec5ce128f248c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:56:23.203610 12309 raft_consensus.cc:1261] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Refusing update from remote peer 324ec59257b9449a86ec5ce128f248c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:56:23.204661 12640 consensus_queue.cc:1035] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:56:23.205027 12647 consensus_queue.cc:1035] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:56:23.217945 12649 mvcc.cc:205] Tried to move back new op lower bound from 7334150894389391360 to 7334150894180061184. Current Snapshot: MvccSnapshot[applied={T|T < 7334150894389391360}]
I20260928 02:56:23.232738 12169 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:56:23.233078 12169 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52104:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:56:23.236936 12169 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Sending DeleteTablet for 3 replicas of tablet 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:23.238324 12289 tablet_service.cc:1514] Processing DeleteTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:47300
I20260928 02:56:23.238507 12421 tablet_service.cc:1514] Processing DeleteTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:33312
I20260928 02:56:23.238641 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12206
I20260928 02:56:23.240139 12552 tablet_service.cc:1514] Processing DeleteTablet for tablet 598e90479c0a4ad7803b153e1bf7e97f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:56:23 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:56448
I20260928 02:56:23.252592 12661 tablet_replica.cc:330] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: stopping tablet replica
I20260928 02:56:23.253118 12661 raft_consensus.cc:2229] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:56:23.253703 12661 raft_consensus.cc:2258] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:23.254626 12659 tablet_replica.cc:330] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: stopping tablet replica
I20260928 02:56:23.255456 12659 raft_consensus.cc:2229] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:56:23.259888 12661 ts_tablet_manager.cc:1905] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:56:23.260221 12659 raft_consensus.cc:2258] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:23.260963 12660 tablet_replica.cc:330] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: stopping tablet replica
I20260928 02:56:23.261497 12660 raft_consensus.cc:2229] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:56:23.262245 12660 raft_consensus.cc:2258] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:56:23.265123 12659 ts_tablet_manager.cc:1905] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:56:23.266186 12628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.239712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.27.229.193:36789)
0928 02:56:23.239934 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.239943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.239966 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:56:23.240037 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.240045 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.240157 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.240283 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:23.242719 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.244417 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.248324 (+ 3907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.249341 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.249590 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.256096 (+ 6506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:23.256119 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:23.256145 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:23.256217 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:23.261443 (+ 5226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.262541 (+ 1098us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.262550 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.262554 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.262725 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.263289 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.263295 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.263301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.263780 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:23.264120 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.265530 (+ 1410us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.265794 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:56:23.266949 12349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:23.268144 12661 ts_tablet_manager.cc:1918] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:56:23.268357 12661 log.cc:1196] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:23.268899 12661 ts_tablet_manager.cc:1939] T 598e90479c0a4ad7803b153e1bf7e97f P b7dd8f580e4b4d98a2fe4b97c12b8d1a: Deleting consensus metadata
I20260928 02:56:23.270053 12660 ts_tablet_manager.cc:1905] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:56:23.270969 12154 catalog_manager.cc:4841] TS b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): tablet 598e90479c0a4ad7803b153e1bf7e97f (table test-table [id=138761cdc64348ddbe381d95d41d9e43]) successfully deleted
I20260928 02:56:23.279755 12660 ts_tablet_manager.cc:1918] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:56:23.279963 12660 log.cc:1196] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:23.280493 12660 ts_tablet_manager.cc:1939] T 598e90479c0a4ad7803b153e1bf7e97f P 324ec59257b9449a86ec5ce128f248c0: Deleting consensus metadata
I20260928 02:56:23.282260 12153 catalog_manager.cc:4841] TS 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): tablet 598e90479c0a4ad7803b153e1bf7e97f (table test-table [id=138761cdc64348ddbe381d95d41d9e43]) successfully deleted
I20260928 02:56:23.397182 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12206
W20260928 02:56:23.415700 12153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.229.193:36789 (error 108)
I20260928 02:56:23.415851 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12358
I20260928 02:56:23.418370 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.417521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47324)
0928 02:56:23.418008 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.418145 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":248,"threads_started":1}
W20260928 02:56:23.418659 12153 catalog_manager.cc:4584] TS 2c69d4aa50b24edf82d8dc7046593ea6 (127.27.229.193:36789): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 598e90479c0a4ad7803b153e1bf7e97f: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
I20260928 02:56:23.434063 12631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.416783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33338 (local address 127.27.229.194:42687)
0928 02:56:23.417274 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.417280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.417301 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:56:23.417369 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.417376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.417462 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.417571 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:23.419255 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.420452 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.421932 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.422717 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.422975 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.425717 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:23.425745 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:23.425766 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:23.425819 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:23.430099 (+ 4280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.431556 (+ 1457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.431566 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.431570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.431781 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.432184 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.432189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.432192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.432752 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:23.433067 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.433413 (+ 346us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.433679 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260928 02:56:23.434998 12481 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:23.472391 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.471869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47332)
0928 02:56:23.472100 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.472210 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:56:23.473455 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.473117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47338)
0928 02:56:23.473257 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.473329 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 02:56:23.534358 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.533856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47344)
0928 02:56:23.534098 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.534192 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 02:56:23.535578 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.535185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47356)
0928 02:56:23.535367 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.535441 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:56:23.553933 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12358
I20260928 02:56:23.573043 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12489
I20260928 02:56:23.588382 12646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.573868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56464 (local address 127.27.229.195:45979)
0928 02:56:23.574058 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:23.574063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:23.574082 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:56:23.574240 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:23.574246 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:23.574315 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:23.574400 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:23.575858 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.576577 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.577886 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:23.578274 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:23.578376 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:23.581248 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:23.581270 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:23.581299 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:23.581350 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:23.585237 (+ 3887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.586054 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.586061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.586064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.586216 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:23.586656 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:23.586660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:23.586663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:23.587302 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:23.587550 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:23.587870 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:56:23.588130 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:56:23.589064 12612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:23.616648 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.616146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47370)
0928 02:56:23.616383 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.616497 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 02:56:23.617899 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.617426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47374)
0928 02:56:23.617675 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.617764 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260928 02:56:23.683383 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12489
I20260928 02:56:23.700644 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12136
I20260928 02:56:23.701829 12199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:56:23.820472 12664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:23.819426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:36789 (local address 127.0.0.1:47384)
0928 02:56:23.820070 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:23.820237 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:36789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 02:56:23.820655 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12136
I20260928 02:56:43.843478 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--webserver_interface=127.27.229.254
--webserver_port=42795
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:56:44.197451 12668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:44.197754 12668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:44.197912 12668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:44.205459 12668 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 02:56:44.205577 12668 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 02:56:44.205647 12668 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 02:56:44.205734 12668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:56:44.205829 12668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:44.205941 12668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:44.206017 12668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:56:44.206110 12668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:56:44.216426 12668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=42795
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:44.217819 12668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:44.219269 12668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:44.231578 12673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:44.231602 12676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:44.231539 12674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:44.232674 12668 server_base.cc:1034] running on GCE node
I20260928 02:56:44.233496 12668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:44.235109 12668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:44.236382 12668 hybrid_clock.cc:648] HybridClock initialized: now 1790564204236323 us; error 33 us; skew 500 ppm
I20260928 02:56:44.236754 12668 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:44 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564204, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:44.240162 12668 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:56:44.241794 12668 webserver.cc:460] Webserver started at http://127.27.229.254:42795/ using document root <none> and password file <none>
I20260928 02:56:44.242522 12668 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:44.242656 12668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:44.248545 12668 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 02:56:44.251526 12683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:44.253082 12668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:56:44.253338 12668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "c9de88825dbf4554b6ae456623a68c20"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "799327b22bb4ef338fc4af846bde8253"
server_key_iv: "3282bb88e8d1a2cf5959d49191a6326b"
server_key_version: "encryptionkey@0"
I20260928 02:56:44.254352 12668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:44.276145 12668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:44.280972 12668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:44.281287 12668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:44.301430 12668 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:35261
I20260928 02:56:44.301470 12734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:35261 every 8 connection(s)
I20260928 02:56:44.303083 12668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:56:44.309101 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12668
I20260928 02:56:44.310165 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:36789
--local_ip_for_outbound_sockets=127.27.229.193
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=42459
--webserver_interface=127.27.229.193
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:56:44.310982 12735 sys_catalog.cc:263] Verifying existing consensus state
I20260928 02:56:44.316751 12735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap starting.
I20260928 02:56:44.344601 12735 log.cc:824] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:44.361927 12735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:56:44.362529 12735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Bootstrap complete.
I20260928 02:56:44.372113 12735 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:44.373243 12735 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Initialized, Role: FOLLOWER
I20260928 02:56:44.373910 12735 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:44.374148 12735 raft_consensus.cc:388] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:56:44.374264 12735 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:56:44.374456 12735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 2 FOLLOWER]: Advancing to term 3
I20260928 02:56:44.378368 12735 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:44.378962 12735 leader_election.cc:302] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c9de88825dbf4554b6ae456623a68c20; no voters:
I20260928 02:56:44.379700 12735 leader_election.cc:288] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 02:56:44.379850 12739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 02:56:44.381173 12739 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [term 3 LEADER]: Becoming Leader. State: Replica: c9de88825dbf4554b6ae456623a68c20, State: Running, Role: LEADER
I20260928 02:56:44.381721 12739 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } }
I20260928 02:56:44.382020 12735 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:56:44.384577 12741 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9de88825dbf4554b6ae456623a68c20. Latest consensus state: current_term: 3 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:56:44.384601 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c9de88825dbf4554b6ae456623a68c20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c9de88825dbf4554b6ae456623a68c20" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35261 } } }
I20260928 02:56:44.385030 12741 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:56:44.385037 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20 [sys.catalog]: This master's current role is: LEADER
I20260928 02:56:44.387462 12744 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:56:44.396579 12744 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:56:44.398236 12744 catalog_manager.cc:1259] Loaded cluster ID: 58f040dd68014adc8713415808340d6f
I20260928 02:56:44.398424 12744 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:56:44.402676 12744 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:56:44.405207 12744 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Loaded TSK: 0
I20260928 02:56:44.405644 12744 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c9de88825dbf4554b6ae456623a68c20: Loaded TSK: 1
F20260928 02:56:44.406944 12744 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f12da9419ad google::LogMessage::Fail() at ??:0
@ 0x7f12da94488a google::LogMessage::SendToLog() at ??:0
@ 0x7f12da9414de google::LogMessage::Flush() at ??:0
@ 0x7f12da942a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f12de6a57d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f12de69473e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f12f03701c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f12f036f783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f12f03fb2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f12f03fb0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f12f04455e9 std::function<>::operator()() at ??:0
@ 0x7f12f0373cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f12f0371c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f12f03f9610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f12f03f9439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f12dd58d345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f12dd59091c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f12dd590799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f12dd56beaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f12e08456db start_thread at ??:0
@ 0x7f12d97db61f clone at ??:0
W20260928 02:56:44.718235 12737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:44.718576 12737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:44.718751 12737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:44.726320 12737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:44.726485 12737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:44.726675 12737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:56:44.737421 12737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:36789
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=42459
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:44.738905 12737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:44.740221 12737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:44.754062 12783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:44.754163 12782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:44.754112 12785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:44.759816 12737 server_base.cc:1034] running on GCE node
I20260928 02:56:44.760593 12737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:44.761650 12737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:44.762974 12737 hybrid_clock.cc:648] HybridClock initialized: now 1790564204762929 us; error 37 us; skew 500 ppm
I20260928 02:56:44.763401 12737 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:44 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564204, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:44.772521 12737 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:56:44.775136 12737 webserver.cc:460] Webserver started at http://127.27.229.193:42459/ using document root <none> and password file <none>
I20260928 02:56:44.775947 12737 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:44.776108 12737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:44.782126 12737 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:56:44.785812 12792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:44.787520 12737 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:56:44.787726 12737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "2c69d4aa50b24edf82d8dc7046593ea6"
format_stamp: "Formatted at 2026-09-28 02:55:58 on dist-test-slave-s1kg"
server_key: "a18d53795922bcd0ab0b182d42c06ea7"
server_key_iv: "755b2fdfd755041770604bbd3fbc8d35"
server_key_version: "encryptionkey@0"
I20260928 02:56:44.788885 12737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:44.807479 12737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:44.812311 12737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:44.812602 12737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:44.813665 12737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:44.816030 12799 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260928 02:56:44.822305 12737 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 02:56:44.822489 12737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260928 02:56:44.822664 12737 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 02:56:44.826030 12737 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 02:56:44.826148 12737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260928 02:56:44.826550 12799 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Bootstrap starting.
I20260928 02:56:44.835806 12799 log.cc:824] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:56:44.852985 12799 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 02:56:44.853681 12799 tablet_bootstrap.cc:492] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Bootstrap complete.
I20260928 02:56:44.854171 12799 ts_tablet_manager.cc:1397] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent bootstrapping tablet: real 0.028s user 0.021s sys 0.004s
I20260928 02:56:44.865053 12799 raft_consensus.cc:348] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:44.866664 12799 raft_consensus.cc:729] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c69d4aa50b24edf82d8dc7046593ea6, State: Initialized, Role: FOLLOWER
I20260928 02:56:44.867712 12799 consensus_queue.cc:260] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:44.869838 12799 ts_tablet_manager.cc:1428] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260928 02:56:44.873576 12737 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:36789
I20260928 02:56:44.873615 12907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:36789 every 8 connection(s)
I20260928 02:56:44.874802 12737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:56:44.879774 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:44.877724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:55851)
0928 02:56:44.878534 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:44.878759 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":341,"threads_started":1}
W20260928 02:56:44.880525 12908 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:35261 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:35261: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
I20260928 02:56:44.880671 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12737
I20260928 02:56:44.881448 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:42687
--local_ip_for_outbound_sockets=127.27.229.194
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=36233
--webserver_interface=127.27.229.194
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:56:44.881505 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:44.881076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:48467)
0928 02:56:44.881248 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:44.881349 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 02:56:44.882541 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:44.882087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:44237)
0928 02:56:44.882269 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:44.882370 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:56:44.882876 12908 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:56:45.220453 12911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:45.220775 12911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:45.221004 12911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:45.228348 12911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:45.228458 12911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:45.228653 12911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:56:45.238322 12911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:42687
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=36233
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:45.239681 12911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:45.241010 12911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:45.254942 12916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:45.254868 12917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:45.254927 12919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:45.256017 12911 server_base.cc:1034] running on GCE node
I20260928 02:56:45.256798 12911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:45.257813 12911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:45.259027 12911 hybrid_clock.cc:648] HybridClock initialized: now 1790564205258978 us; error 40 us; skew 500 ppm
I20260928 02:56:45.259433 12911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:45 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:45.263197 12911 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:56:45.264792 12911 webserver.cc:460] Webserver started at http://127.27.229.194:36233/ using document root <none> and password file <none>
I20260928 02:56:45.265527 12911 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:45.265689 12911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:45.271313 12911 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:56:45.274288 12926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.276006 12911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:56:45.276218 12911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "324ec59257b9449a86ec5ce128f248c0"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "d0366868a69d78819d04f15e6555bfcd"
server_key_iv: "3478b1b64c48c933ff5aac35b86a1c64"
server_key_version: "encryptionkey@0"
I20260928 02:56:45.277252 12911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:45.297231 12911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:45.301786 12911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:45.302088 12911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:45.303277 12911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:45.304728 12911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:56:45.304863 12911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.305001 12911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:56:45.305089 12911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.339996 12911 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:42687
I20260928 02:56:45.340085 13038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:42687 every 8 connection(s)
I20260928 02:56:45.341933 12911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:56:45.347329 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 12911
I20260928 02:56:45.347338 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.345331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:50605)
0928 02:56:45.346107 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.346375 (+ 268us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":292,"threads_started":1}
W20260928 02:56:45.347962 13039 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:35261 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:35261: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
I20260928 02:56:45.348160 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:45979
--local_ip_for_outbound_sockets=127.27.229.195
--tserver_master_addrs=127.27.229.254:35261
--webserver_port=46311
--webserver_interface=127.27.229.195
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:56:45.348982 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.348594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:36399)
0928 02:56:45.348781 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.348870 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 02:56:45.349886 13041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.349563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:49799)
0928 02:56:45.349694 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.349777 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:56:45.350178 13039 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 02:56:45.693831 13042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:56:45.694242 13042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:56:45.694433 13042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:56:45.702493 13042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:56:45.702625 13042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:56:45.702813 13042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:56:45.713083 13042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:45979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=46311
--tserver_master_addrs=127.27.229.254:35261
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:56:45.714543 13042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:56:45.715938 13042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:56:45.729530 13048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:45.729583 13047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:56:45.729609 13050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:56:45.731137 13042 server_base.cc:1034] running on GCE node
I20260928 02:56:45.731892 13042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:56:45.732914 13042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:56:45.734154 13042 hybrid_clock.cc:648] HybridClock initialized: now 1790564205734103 us; error 47 us; skew 500 ppm
I20260928 02:56:45.734488 13042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:56:45 dist-test-slave-s1kg krb5kdc[11585](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564205, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:56:45.737825 13042 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:56:45.739463 13042 webserver.cc:460] Webserver started at http://127.27.229.195:46311/ using document root <none> and password file <none>
I20260928 02:56:45.740191 13042 fs_manager.cc:359] Metadata directory not provided
I20260928 02:56:45.740347 13042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:56:45.746166 13042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 02:56:45.749233 13057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.751055 13042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 02:56:45.751264 13042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a"
format_stamp: "Formatted at 2026-09-28 02:55:59 on dist-test-slave-s1kg"
server_key: "e151992d0192a90e17edc618a5b83812"
server_key_iv: "650f7fedb5020b5a3a75e112a6eaebc6"
server_key_version: "encryptionkey@0"
I20260928 02:56:45.752396 13042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:56:45.780067 13042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:56:45.785101 13042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:56:45.785421 13042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:56:45.786546 13042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:56:45.788041 13042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:56:45.788180 13042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.788322 13042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:56:45.788411 13042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:56:45.825639 13042 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:45979
I20260928 02:56:45.825681 13169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:45979 every 8 connection(s)
I20260928 02:56:45.827150 13042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:56:45.832392 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.830320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:53529)
0928 02:56:45.831272 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.831531 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":355,"threads_started":1}
W20260928 02:56:45.833305 13170 heartbeater.cc:644] Failed to heartbeat to 127.27.229.254:35261 (0 consecutive failures): Network error: Failed to ping master at 127.27.229.254:35261: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
I20260928 02:56:45.834316 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.833928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:58723)
0928 02:56:45.834079 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.834193 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 02:56:45.835494 13172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.835068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:36199)
0928 02:56:45.835284 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.835370 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 02:56:45.835775 13170 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 02:56:45.836930 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13042
I20260928 02:56:45.884415 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:45.883489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:60179)
0928 02:56:45.884009 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:45.884151 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":280,"threads_started":1}
I20260928 02:56:46.283051 13175 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:46.283393 13175 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:46.285171 13175 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
Sep 28 02:56:46 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564204, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
Sep 28 02:56:46 dist-test-slave-s1kg krb5kdc[11585](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564204, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:56:46.310685 13174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.285535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:45979 (local address 127.27.229.193:55359)
0928 02:56:46.285717 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:46.285763 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:46.286444 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:46.290450 (+ 4006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:46.290459 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:46.291425 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:56:46.292641 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:46.292668 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.294944 (+ 2276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.294950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:46.296456 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:46.296471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.298703 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.298711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:46.298925 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:46.299530 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:56:46.299548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:56:46.301770 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:56:46.306233 (+ 4463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:46.306276 (+ 43us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:46.306289 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:46.306804 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:46.307317 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:46.307323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:46.307328 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:46.307591 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:46.308838 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:56:46.308844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:56:46.309143 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 02:56:46.309556 (+ 413us) client_negotiation.cc:240] Negotiation successful
0928 02:56:46.310314 (+ 758us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":308,"spinlock_wait_cycles":86400}
I20260928 02:56:46.310925 13176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.285708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:44003 (local address 127.27.229.194:42687)
0928 02:56:46.287546 (+ 1838us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:46.287557 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:46.288176 (+ 619us) server_negotiation.cc:407] Connection header received
0928 02:56:46.290275 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:46.290284 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:46.290723 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:46.291041 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:46.292949 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.294579 (+ 1630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.296620 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.297415 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.297577 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:46.301942 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:46.301954 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:46.301979 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:46.302020 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:46.306054 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:46.307697 (+ 1643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:46.307710 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:46.307750 (+ 40us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:46.308020 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:46.308477 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:46.308483 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:46.308486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:46.309163 (+ 677us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:46.309315 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:46.309913 (+ 598us) server_negotiation.cc:299] Negotiation successful
0928 02:56:46.310464 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":1559,"threads_started":1}
I20260928 02:56:46.310928 13178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.285945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:55359 (local address 127.27.229.195:45979)
0928 02:56:46.287288 (+ 1343us) server_negotiation.cc:206] Beginning negotiation
0928 02:56:46.287305 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:56:46.287353 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:56:46.289329 (+ 1976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:56:46.289342 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:56:46.289853 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:56:46.290229 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:56:46.292918 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.294716 (+ 1798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.297788 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.298459 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.298679 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:46.302039 (+ 3360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:56:46.302047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:56:46.302060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:56:46.302105 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:56:46.306055 (+ 3950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:46.306993 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:46.307000 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:46.307006 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:46.307170 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:56:46.307747 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:56:46.307753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:56:46.307763 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:56:46.308636 (+ 873us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:56:46.308860 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 02:56:46.309897 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0928 02:56:46.310455 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":1033,"threads_started":1}
I20260928 02:56:46.310683 13177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.285553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:42687 (local address 127.27.229.193:44003)
0928 02:56:46.288037 (+ 2484us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:46.288061 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:56:46.288178 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:56:46.291062 (+ 2884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:56:46.291068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:56:46.291700 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:56:46.292645 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:46.292680 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.294810 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.294814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:46.296458 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:56:46.296471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:56:46.297632 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:56:46.297637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:56:46.297829 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:56:46.298585 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:56:46.298609 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:56:46.301556 (+ 2947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:56:46.306952 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:46.306957 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:46.306961 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:46.307483 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:46.308176 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:56:46.308181 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:56:46.308184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:56:46.308338 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:56:46.309306 (+ 968us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:56:46.309310 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:56:46.309462 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 02:56:46.309584 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 02:56:46.310171 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2341,"mutex_wait_us":589,"spinlock_wait_cycles":6912,"thread_start_us":1869,"threads_started":1}
W20260928 02:56:46.314953 12793 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
W20260928 02:56:46.315261 12794 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:46.315420 12794 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:46.315893 13175 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:56:46.351845 13179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.350879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:37571)
0928 02:56:46.351438 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:46.351594 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":279,"threads_started":1}
I20260928 02:56:46.837159 13180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.836371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:36889)
0928 02:56:46.836831 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:46.836942 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":248,"threads_started":1}
I20260928 02:56:46.886220 13181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:46.885469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:37981)
0928 02:56:46.885945 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:46.886039 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":241,"threads_started":1}
I20260928 02:56:47.353883 13182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:47.353048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:40273)
0928 02:56:47.353590 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:47.353715 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":286,"threads_started":1}
I20260928 02:56:47.838925 13183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:47.838133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:58557)
0928 02:56:47.838561 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:47.838647 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":240,"threads_started":1}
I20260928 02:56:47.888159 13184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:47.887354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:54271)
0928 02:56:47.887887 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:47.887975 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":267,"threads_started":1}
I20260928 02:56:48.136000 13185 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:48.136246 13185 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:48.136981 13185 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
W20260928 02:56:48.138118 12793 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
W20260928 02:56:48.138319 12794 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:48.138463 12794 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:48.138751 13185 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:56:48.355954 13186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:48.354944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:54979)
0928 02:56:48.355531 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:48.355691 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
I20260928 02:56:48.840847 13187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:48.839987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:40535)
0928 02:56:48.840574 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:48.840673 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":354,"threads_started":1}
I20260928 02:56:48.889963 13188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:48.889153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:47503)
0928 02:56:48.889687 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:48.889786 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":260,"threads_started":1}
I20260928 02:56:49.358464 13189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:49.357055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:56199)
0928 02:56:49.357648 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:49.357794 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":248,"threads_started":1}
I20260928 02:56:49.842792 13190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:49.841971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:43957)
0928 02:56:49.842517 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:49.842605 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":264,"threads_started":1}
I20260928 02:56:49.892081 13191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:49.891259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:49945)
0928 02:56:49.891802 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:49.891889 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":242,"threads_started":1}
I20260928 02:56:50.118232 13192 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:50.118494 13192 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:50.119313 13192 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
W20260928 02:56:50.120509 12794 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
W20260928 02:56:50.120754 12793 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:50.120988 12793 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:50.121313 13192 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:56:50.360322 13193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:50.359544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:36429)
0928 02:56:50.360040 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:50.360128 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260928 02:56:50.844679 13194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:50.843945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:36081)
0928 02:56:50.844416 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:50.844515 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":254,"threads_started":1}
I20260928 02:56:50.894096 13195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:50.893251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:50071)
0928 02:56:50.893725 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:50.893887 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":240,"threads_started":1}
I20260928 02:56:51.362437 13196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:51.361313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:57783)
0928 02:56:51.362045 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:51.362167 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":338,"threads_started":1}
I20260928 02:56:51.846554 13197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:51.845781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:48947)
0928 02:56:51.846276 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:51.846361 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":272,"threads_started":1}
I20260928 02:56:51.896307 13198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:51.895413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:36311)
0928 02:56:51.895941 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:51.896039 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":255,"threads_started":1}
I20260928 02:56:52.364516 13199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:52.363594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:52291)
0928 02:56:52.364215 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:52.364321 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":307,"threads_started":1}
I20260928 02:56:52.829958 13200 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:52.830212 13200 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:52.831060 13200 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
W20260928 02:56:52.832167 12794 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
W20260928 02:56:52.832610 12793 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:52.832722 12793 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:52.832988 13200 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:56:52.848752 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:52.847782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:48561)
0928 02:56:52.848424 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:52.848514 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":312,"threads_started":1}
I20260928 02:56:52.898068 13202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:52.897208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:52491)
0928 02:56:52.897741 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:52.897879 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":279,"threads_started":1}
I20260928 02:56:53.366700 13203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:53.365676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:49565)
0928 02:56:53.366240 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:53.366438 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":328,"threads_started":1}
I20260928 02:56:53.850796 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:53.849867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:59557)
0928 02:56:53.850476 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:53.850596 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":317,"threads_started":1}
I20260928 02:56:53.900127 13205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:53.899203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:60305)
0928 02:56:53.899854 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:53.899940 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":321,"threads_started":1}
I20260928 02:56:54.368721 13206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:54.367948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:58631)
0928 02:56:54.368461 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:54.368554 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":261,"threads_started":1}
I20260928 02:56:54.852595 13207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:54.851823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:49091)
0928 02:56:54.852342 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:54.852433 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":250,"threads_started":1}
I20260928 02:56:54.902119 13208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:54.901317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:43567)
0928 02:56:54.901833 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:54.901929 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":267,"threads_started":1}
I20260928 02:56:55.370965 13209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:55.369935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:49665)
0928 02:56:55.370590 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:55.370684 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":384,"threads_started":1}
I20260928 02:56:55.795959 13210 raft_consensus.cc:482] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:56:55.796206 13210 raft_consensus.cc:504] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c69d4aa50b24edf82d8dc7046593ea6" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 36789 } } peers { permanent_uuid: "324ec59257b9449a86ec5ce128f248c0" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 42687 } } peers { permanent_uuid: "b7dd8f580e4b4d98a2fe4b97c12b8d1a" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 45979 } }
I20260928 02:56:55.797025 13210 leader_election.cc:288] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687), b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979)
W20260928 02:56:55.798055 12794 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b7dd8f580e4b4d98a2fe4b97c12b8d1a (127.27.229.195:45979): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
W20260928 02:56:55.798257 12793 leader_election.cc:341] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 324ec59257b9449a86ec5ce128f248c0 (127.27.229.194:42687): Not found: Tablet not found: 598e90479c0a4ad7803b153e1bf7e97f
I20260928 02:56:55.798405 12793 leader_election.cc:302] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2c69d4aa50b24edf82d8dc7046593ea6; no voters: 324ec59257b9449a86ec5ce128f248c0, b7dd8f580e4b4d98a2fe4b97c12b8d1a
I20260928 02:56:55.798731 13210 raft_consensus.cc:2726] T 598e90479c0a4ad7803b153e1bf7e97f P 2c69d4aa50b24edf82d8dc7046593ea6 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 02:56:55.854473 13211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:55.853637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:42725)
0928 02:56:55.854129 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:55.854266 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":274,"threads_started":1}
I20260928 02:56:55.904176 13212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:55.903284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:38845)
0928 02:56:55.903826 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:55.903945 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":277,"threads_started":1}
I20260928 02:56:56.372896 13213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:56.372044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:49903)
0928 02:56:56.372597 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:56.372707 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":278,"threads_started":1}
I20260928 02:56:56.856520 13214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:56.855689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:45697)
0928 02:56:56.856233 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:56.856359 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":272,"threads_started":1}
I20260928 02:56:56.905901 13215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:56.905135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:54849)
0928 02:56:56.905626 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:56.905721 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":277,"threads_started":1}
I20260928 02:56:57.374778 13216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:57.374002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:44637)
0928 02:56:57.374507 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:57.374597 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":289,"threads_started":1}
I20260928 02:56:57.858358 13217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:57.857542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:38931)
0928 02:56:57.858062 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:57.858164 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":279,"threads_started":1}
I20260928 02:56:57.907950 13218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:57.907069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:39337)
0928 02:56:57.907626 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:57.907755 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":296,"threads_started":1}
I20260928 02:56:58.376857 13219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:58.375952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:50657)
0928 02:56:58.376541 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:58.376651 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":323,"threads_started":1}
I20260928 02:56:58.860240 13220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:58.859460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:52953)
0928 02:56:58.859955 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:58.860074 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":233,"threads_started":1}
I20260928 02:56:58.910019 13221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:58.909116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:54735)
0928 02:56:58.909732 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:58.909816 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":279,"threads_started":1}
I20260928 02:56:59.378782 13222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:59.378009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:50239)
0928 02:56:59.378485 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:59.378570 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":252,"threads_started":1}
I20260928 02:56:59.862001 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:59.861184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:32789)
0928 02:56:59.861681 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:59.861793 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":274,"threads_started":1}
I20260928 02:56:59.911983 13224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:56:59.911210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.193:53111)
0928 02:56:59.911718 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 02:56:59.911812 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":251,"threads_started":1}
I20260928 02:57:00.380934 13225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:00.380024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.194:60563)
0928 02:57:00.380643 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:00.380767 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":404,"threads_started":1}
I20260928 02:57:00.842850 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12737
I20260928 02:57:00.861066 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:00.844194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.27.229.193:36789)
0928 02:57:00.844866 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:00.844873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:00.844899 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:57:00.844962 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:00.844968 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:00.845181 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:00.845437 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:00.847419 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:00.848575 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:00.850026 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:00.850657 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:00.850821 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:00.853707 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:00.853721 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:00.853737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:00.853782 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:00.857688 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:00.858545 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:00.858554 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:00.858556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:00.858705 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:00.859302 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:00.859311 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:00.859314 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:00.859716 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:00.860043 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:00.860444 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 02:57:00.860732 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":333,"threads_started":1}
I20260928 02:57:00.862558 12903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:00.867208 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:00.863414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35261 (local address 127.27.229.195:59265)
0928 02:57:00.866854 (+ 3440us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:00.867029 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.254:35261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3268,"thread_start_us":3128,"threads_started":1}
I20260928 02:57:00.953994 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12737
I20260928 02:57:00.973126 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 12911
I20260928 02:57:00.987011 13230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:00.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.27.229.194:42687)
0928 02:57:00.974437 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:00.974441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:00.974457 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:57:00.974511 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:00.974514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:00.974573 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:00.974692 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:00.976098 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:00.976878 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:00.978099 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:00.978540 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:00.978682 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:00.981457 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:00.981462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:00.981466 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:00.981500 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:00.984238 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:00.984953 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:00.984960 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:00.984964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:00.985089 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:00.985451 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:00.985457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:00.985460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:00.985732 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:00.986002 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:00.986317 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 02:57:00.986542 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":264,"threads_started":1}
I20260928 02:57:00.987699 13034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:01.072621 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 12911
I20260928 02:57:01.090971 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13042
I20260928 02:57:01.105475 13232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:01.091938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59390 (local address 127.27.229.195:45979)
0928 02:57:01.092526 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:01.092531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:01.092548 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:57:01.092602 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:01.092605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:01.092678 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:01.092806 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:01.094256 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:01.095182 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:01.096417 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:01.096892 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:01.097035 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:01.099839 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:01.099844 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:01.099848 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:01.099883 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:01.102573 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:01.103358 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:01.103364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:01.103368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:01.103530 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:01.103931 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:01.103940 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:01.103943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:01.104229 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:01.104526 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:01.104829 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 02:57:01.105071 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":332,"threads_started":1}
I20260928 02:57:01.106349 13165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:01.192755 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13042
2026-09-28T02:57:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64884 ms)
[----------] 30 tests from SecurityITest (305507 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:01 dist-test-slave-s1kg krb5kdc[13236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:01 dist-test-slave-s1kg krb5kdc[13236](info): set up 2 sockets
Sep 28 02:57:01 dist-test-slave-s1kg krb5kdc[13236](info): commencing operation
krb5kdc: starting...
W20260928 02:57:02.311178 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:02 dist-test-slave-s1kg krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:02.576617 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42895
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:42895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:02.923418 13249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:02.923816 13249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:02.924009 13249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:02.931778 13249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:02.931967 13249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:02.932080 13249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:02.932194 13249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:02.932278 13249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:02.942518 13249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:42895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:42895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:02.944013 13249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:02.945422 13249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:02.958055 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:02.958218 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:02.958240 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:02.964526 13249 server_base.cc:1034] running on GCE node
I20260928 02:57:02.965544 13249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:02.967237 13249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:02.968526 13249 hybrid_clock.cc:648] HybridClock initialized: now 1790564222968468 us; error 60 us; skew 500 ppm
I20260928 02:57:02.968973 13249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:02 dist-test-slave-s1kg krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:02.972661 13249 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:02.974815 13249 webserver.cc:460] Webserver started at http://127.27.229.254:34649/ using document root <none> and password file <none>
I20260928 02:57:02.975720 13249 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:02.975891 13249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:02.976219 13249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:02.979665 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "88715cc2f69c431c8f43f82e30200e8c"
format_stamp: "Formatted at 2026-09-28 02:57:02 on dist-test-slave-s1kg"
server_key: "12c4395d8940d961cbf23480c3dab2f0"
server_key_iv: "1fdf33dc92d2be53463a747c721725f7"
server_key_version: "encryptionkey@0"
I20260928 02:57:02.980561 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "88715cc2f69c431c8f43f82e30200e8c"
format_stamp: "Formatted at 2026-09-28 02:57:02 on dist-test-slave-s1kg"
server_key: "12c4395d8940d961cbf23480c3dab2f0"
server_key_iv: "1fdf33dc92d2be53463a747c721725f7"
server_key_version: "encryptionkey@0"
I20260928 02:57:02.986372 13249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:57:02.990502 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:02.992497 13249 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260928 02:57:02.992724 13249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "88715cc2f69c431c8f43f82e30200e8c"
format_stamp: "Formatted at 2026-09-28 02:57:02 on dist-test-slave-s1kg"
server_key: "12c4395d8940d961cbf23480c3dab2f0"
server_key_iv: "1fdf33dc92d2be53463a747c721725f7"
server_key_version: "encryptionkey@0"
I20260928 02:57:02.992965 13249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:03.019075 13249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:03.023779 13249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:03.024106 13249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:03.044639 13249 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:42895
I20260928 02:57:03.044663 13315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:42895 every 8 connection(s)
I20260928 02:57:03.046108 13249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:03.050470 13316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:03.053567 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13249
I20260928 02:57:03.053798 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:03.054463 28567 external_mini_cluster.cc:1428] Setting key 38ee1377a36af34be1d81eaae9f098da
I20260928 02:57:03.063436 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: Bootstrap starting.
I20260928 02:57:03.067327 13316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:03.068531 13316 log.cc:824] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:57:03 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:03.072121 13316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: No bootstrap required, opened a new log
I20260928 02:57:03.080551 13319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:03.056632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.27.229.254:42895)
0928 02:57:03.057477 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:03.057489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:03.057533 (+ 44us) server_negotiation.cc:407] Connection header received
0928 02:57:03.059412 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:03.059438 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:03.059889 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:03.060322 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:03.061903 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.063350 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.064864 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.065613 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.065841 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:03.070919 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:03.070932 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:03.070952 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:03.071003 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:03.075297 (+ 4294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:03.076181 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:03.076193 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:03.076202 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:03.076440 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:03.076909 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:03.076914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:03.076917 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:03.077702 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:03.078018 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:03.078474 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 02:57:03.079020 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":389,"threads_started":1}
I20260928 02:57:03.083230 13316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } }
I20260928 02:57:03.083765 13316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:03.083913 13316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88715cc2f69c431c8f43f82e30200e8c, State: Initialized, Role: FOLLOWER
I20260928 02:57:03.084627 13316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } }
I20260928 02:57:03.084908 13316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:03.085059 13316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:03.085242 13316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:03.088151 13316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } }
I20260928 02:57:03.088783 13316 leader_election.cc:302] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88715cc2f69c431c8f43f82e30200e8c; no voters:
I20260928 02:57:03.089463 13316 leader_election.cc:288] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:03.089643 13321 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:03.090430 13321 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [term 1 LEADER]: Becoming Leader. State: Replica: 88715cc2f69c431c8f43f82e30200e8c, State: Running, Role: LEADER
I20260928 02:57:03.091156 13321 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } }
I20260928 02:57:03.091779 13316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:03.095589 13323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88715cc2f69c431c8f43f82e30200e8c. Latest consensus state: current_term: 1 leader_uuid: "88715cc2f69c431c8f43f82e30200e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } } }
I20260928 02:57:03.096119 13323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:03.095511 13322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88715cc2f69c431c8f43f82e30200e8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88715cc2f69c431c8f43f82e30200e8c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 42895 } } }
I20260928 02:57:03.097820 13322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:03.101485 13330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:03.108028 13330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:03.119930 13330 catalog_manager.cc:1347] Generated new cluster ID: 77a0638f57f9460ca2b602272e28cf20
I20260928 02:57:03.120114 13330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:03.136257 13330 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:03.137739 13330 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:03.170953 13330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: Generated new TSK 0
I20260928 02:57:03.171993 13330 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:57:03.247430 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:03.606596 13342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:03.607009 13342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:03.607132 13342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:03.615784 13342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:03.615912 13342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:03.616031 13342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:03.627240 13342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:03.628599 13342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:03.630388 13342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:03.645502 13347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:03.645510 13348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:03.645495 13350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:03.646382 13342 server_base.cc:1034] running on GCE node
I20260928 02:57:03.647149 13342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:03.648231 13342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:03.649506 13342 hybrid_clock.cc:648] HybridClock initialized: now 1790564223649448 us; error 73 us; skew 500 ppm
I20260928 02:57:03.649870 13342 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:03 dist-test-slave-s1kg krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564223, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:03.653596 13342 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:57:03.655325 13342 webserver.cc:460] Webserver started at http://127.27.229.193:37889/ using document root <none> and password file <none>
I20260928 02:57:03.656060 13342 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:03.656180 13342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:03.656479 13342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:03.659793 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "efbbf04d6049427ab62dc0d0f2b34c18"
format_stamp: "Formatted at 2026-09-28 02:57:03 on dist-test-slave-s1kg"
server_key: "82cd65b2c2ec3f20642b6ab8a38a35d5"
server_key_iv: "c44c254925342a479052495f557326b9"
server_key_version: "encryptionkey@0"
I20260928 02:57:03.660607 13342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "efbbf04d6049427ab62dc0d0f2b34c18"
format_stamp: "Formatted at 2026-09-28 02:57:03 on dist-test-slave-s1kg"
server_key: "82cd65b2c2ec3f20642b6ab8a38a35d5"
server_key_iv: "c44c254925342a479052495f557326b9"
server_key_version: "encryptionkey@0"
I20260928 02:57:03.666977 13342 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260928 02:57:03.671129 13357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:03.672950 13342 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260928 02:57:03.673120 13342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "efbbf04d6049427ab62dc0d0f2b34c18"
format_stamp: "Formatted at 2026-09-28 02:57:03 on dist-test-slave-s1kg"
server_key: "82cd65b2c2ec3f20642b6ab8a38a35d5"
server_key_iv: "c44c254925342a479052495f557326b9"
server_key_version: "encryptionkey@0"
I20260928 02:57:03.673344 13342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:03.695997 13342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:03.700911 13342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:03.701249 13342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:03.702407 13342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:03.704021 13342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:03.704100 13342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:03.704250 13342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:03.704312 13342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:03.739977 13342 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:34011
I20260928 02:57:03.740090 13469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:34011 every 8 connection(s)
I20260928 02:57:03.741350 13342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:03.744091 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13342
I20260928 02:57:03.744342 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:03.745132 28567 external_mini_cluster.cc:1428] Setting key a8e74f98e8c6150a4e01409289a01fff
Sep 28 02:57:03 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564223, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:03.766402 13472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:03.744112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:42301 (local address 127.27.229.254:42895)
0928 02:57:03.744714 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:03.744721 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:03.745906 (+ 1185us) server_negotiation.cc:407] Connection header received
0928 02:57:03.748444 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:03.748449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:03.748523 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:03.748619 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:03.750824 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.751654 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.754334 (+ 2680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.754813 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.755009 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:03.758886 (+ 3877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:03.758897 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:03.758904 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:03.758956 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:03.762635 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:03.763588 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:03.763596 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:03.763602 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:03.763760 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:03.764236 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:03.764244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:03.764250 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:03.764665 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:03.764825 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:03.765630 (+ 805us) server_negotiation.cc:299] Negotiation successful
0928 02:57:03.765904 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":324,"threads_started":1}
I20260928 02:57:03.767269 13473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:03.744772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42895 (local address 127.27.229.193:42301)
0928 02:57:03.745712 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:03.745765 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:03.748066 (+ 2301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:03.748929 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:03.748943 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:03.749765 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:03.750617 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:03.750646 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.751841 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.751849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:03.754146 (+ 2297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:03.754166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:03.755128 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:03.755137 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:03.755489 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:03.756316 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:03.756360 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:03.758594 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:03.762927 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:03.762936 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:03.762952 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:03.763450 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:03.763907 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:03.763911 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:03.763913 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:03.764102 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:03.764814 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:03.764821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:03.765206 (+ 385us) client_negotiation.cc:769] Sending connection context
0928 02:57:03.765458 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 02:57:03.765903 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":361,"threads_started":1}
I20260928 02:57:03.769153 13470 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42895
I20260928 02:57:03.769479 13470 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:03.770236 13470 heartbeater.cc:511] Master 127.27.229.254:42895 requested a full tablet report, sending...
I20260928 02:57:03.772585 13281 ts_manager.cc:194] Registered new tserver with Master: efbbf04d6049427ab62dc0d0f2b34c18 (127.27.229.193:34011)
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
I20260928 02:57:03.774161 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:42301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:57:03.807328 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:04.148895 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:04.149211 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:04.149312 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:04.156845 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:04.156973 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:04.157088 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:04.167214 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:04.168540 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:04.169804 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:04.183516 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:04.183634 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:04.184259 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:04.185348 13476 server_base.cc:1034] running on GCE node
I20260928 02:57:04.186091 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:04.187186 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:04.188441 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790564224188356 us; error 43 us; skew 500 ppm
I20260928 02:57:04.188910 13476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:04 dist-test-slave-s1kg krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:04.192433 13476 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:57:04.194067 13476 webserver.cc:460] Webserver started at http://127.27.229.194:39717/ using document root <none> and password file <none>
I20260928 02:57:04.194906 13476 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:04.195065 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:04.195499 13476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:04.198712 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "c27d27ea867d6f2eb9e4b9412a0e88ef"
server_key_iv: "92911921c39d643f5bf412fb766cdc56"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.199679 13476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "c27d27ea867d6f2eb9e4b9412a0e88ef"
server_key_iv: "92911921c39d643f5bf412fb766cdc56"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.205957 13476 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:57:04.209919 13491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.211789 13476 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:57:04.212019 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "c27d27ea867d6f2eb9e4b9412a0e88ef"
server_key_iv: "92911921c39d643f5bf412fb766cdc56"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.212260 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:04.229283 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:04.234581 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:04.235002 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:04.236137 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:04.237663 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:04.237779 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.237953 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:04.238054 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.280054 13476 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:41073
I20260928 02:57:04.280186 13603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:41073 every 8 connection(s)
I20260928 02:57:04.281513 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:04.283314 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13476
I20260928 02:57:04.283501 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:04.284032 28567 external_mini_cluster.cc:1428] Setting key e8570dc0ac57450493ce936b0024a2c5
Sep 28 02:57:04 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:04.307103 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.284734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:33525 (local address 127.27.229.254:42895)
0928 02:57:04.285209 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.285214 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.286905 (+ 1691us) server_negotiation.cc:407] Connection header received
0928 02:57:04.289166 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.289172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.289278 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.289399 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:04.291057 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.291862 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.293532 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.294108 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.294250 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.299266 (+ 5016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:04.299275 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:04.299281 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:04.299332 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:04.303108 (+ 3776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.304254 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.304259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.304261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.304452 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.305014 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.305018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.305020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.305304 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:04.305430 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.306416 (+ 986us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.306726 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":247,"threads_started":1}
I20260928 02:57:04.308555 13607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.285430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42895 (local address 127.27.229.194:33525)
0928 02:57:04.286396 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:04.286459 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:04.288781 (+ 2322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:04.289649 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:04.289658 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:04.290200 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:04.290803 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.290821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.292030 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.292035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.293379 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.293391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.294355 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.294361 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.294570 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.295773 (+ 1203us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:04.295808 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:04.298971 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:04.303304 (+ 4333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:04.303318 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:04.303341 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:04.304043 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:04.304588 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:04.304596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:04.304602 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:04.304885 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:04.305448 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:04.305460 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:04.305982 (+ 522us) client_negotiation.cc:769] Sending connection context
0928 02:57:04.306293 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 02:57:04.306957 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":376,"threads_started":1}
I20260928 02:57:04.310186 13604 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42895
I20260928 02:57:04.310545 13604 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:57:04.311566 13604 heartbeater.cc:511] Master 127.27.229.254:42895 requested a full tablet report, sending...
I20260928 02:57:04.313153 13281 ts_manager.cc:194] Registered new tserver with Master: f9a4b8c8ef85441b91e5f67d5ec877e6 (127.27.229.194:41073)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
I20260928 02:57:04.314477 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:33525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:57:04.343907 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:43417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:04.687871 13610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:04.688166 13610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:04.688263 13610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:04.695837 13610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:04.695957 13610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:04.696127 13610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:04.706720 13610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:43417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:42895
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:04.708004 13610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:04.709393 13610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:04.723500 13618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:04.723460 13615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:04.724133 13616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:04.724363 13610 server_base.cc:1034] running on GCE node
I20260928 02:57:04.725005 13610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:04.726061 13610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:04.727317 13610 hybrid_clock.cc:648] HybridClock initialized: now 1790564224727269 us; error 40 us; skew 500 ppm
I20260928 02:57:04.727663 13610 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:04 dist-test-slave-s1kg krb5kdc[13236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:04.735796 13610 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:57:04.737552 13610 webserver.cc:460] Webserver started at http://127.27.229.195:44219/ using document root <none> and password file <none>
I20260928 02:57:04.738376 13610 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:04.738531 13610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:04.738935 13610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:04.741935 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "447bd3b36cdb44c5a585fef881790602"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "842252de08daa013d8af7ab639f51859"
server_key_iv: "b1268a077216ffd61577491ed3a5d60e"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.742690 13610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "447bd3b36cdb44c5a585fef881790602"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "842252de08daa013d8af7ab639f51859"
server_key_iv: "b1268a077216ffd61577491ed3a5d60e"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.748004 13610 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:57:04.751683 13625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.753401 13610 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:04.753618 13610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "447bd3b36cdb44c5a585fef881790602"
format_stamp: "Formatted at 2026-09-28 02:57:04 on dist-test-slave-s1kg"
server_key: "842252de08daa013d8af7ab639f51859"
server_key_iv: "b1268a077216ffd61577491ed3a5d60e"
server_key_version: "encryptionkey@0"
I20260928 02:57:04.753868 13610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:04.772325 13610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:04.776593 13610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:04.776935 13610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:04.777931 13470 heartbeater.cc:503] Master 127.27.229.254:42895 was elected leader, sending a full tablet report...
I20260928 02:57:04.778012 13610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:04.780467 13610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:04.780572 13610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.780722 13610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:04.780794 13610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:04.816743 13610 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:38207
I20260928 02:57:04.816828 13737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:38207 every 8 connection(s)
I20260928 02:57:04.818111 13610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:04.819907 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13610
I20260928 02:57:04.820107 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:04.820725 28567 external_mini_cluster.cc:1428] Setting key ae0878f422f08a39f285509c13df3273
Sep 28 02:57:04 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564224, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:04.839176 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.821103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:33635 (local address 127.27.229.254:42895)
0928 02:57:04.821700 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.821708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.822790 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 02:57:04.824737 (+ 1947us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.824741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.824825 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.824917 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:04.826401 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.827290 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.828804 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.829211 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.829328 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.832825 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:04.832846 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:04.832850 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:04.832885 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:04.836077 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.836955 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.836959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.836961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.837079 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.837493 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.837496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.837498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.837774 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:04.837918 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.838656 (+ 738us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.838898 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
I20260928 02:57:04.840225 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.821778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:42895 (local address 127.27.229.195:33635)
0928 02:57:04.822630 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:04.822675 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:04.824471 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:04.825096 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:04.825104 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:04.825624 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:04.826188 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.826204 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.827440 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.827444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.828654 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.828666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.829429 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.829434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.829620 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.830234 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:04.830291 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:04.832605 (+ 2314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:04.836233 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:04.836244 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:04.836263 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:04.836846 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:04.837178 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:04.837181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:04.837183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:04.837375 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:04.837926 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:04.837933 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:04.838299 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 02:57:04.838559 (+ 260us) client_negotiation.cc:240] Negotiation successful
0928 02:57:04.839062 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":364,"threads_started":1}
I20260928 02:57:04.841941 13738 heartbeater.cc:347] Connected to a master server at 127.27.229.254:42895
I20260928 02:57:04.842262 13738 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:04.843071 13738 heartbeater.cc:511] Master 127.27.229.254:42895 requested a full tablet report, sending...
I20260928 02:57:04.844612 13281 ts_manager.cc:194] Registered new tserver with Master: 447bd3b36cdb44c5a585fef881790602 (127.27.229.195:38207)
I20260928 02:57:04.845472 13281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:33635
I20260928 02:57:04.849761 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:04.868533 13740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.855035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.27.229.254:42895)
0928 02:57:04.855235 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.855239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.855588 (+ 349us) server_negotiation.cc:407] Connection header received
0928 02:57:04.855719 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.855723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.855824 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.855930 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:04.857616 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.858426 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.859955 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.860347 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.860465 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.862884 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:04.862889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:04.862892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:04.862929 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:04.865654 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.866370 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.866374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.866377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.866510 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:04.867069 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:04.867074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:04.867076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:04.867473 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:04.867676 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.868081 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.868263 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:57:04.872910 13281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:04.880162 13281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:04.919324 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.899361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33692 (local address 127.27.229.195:38207)
0928 02:57:04.910036 (+ 10675us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.910041 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.910071 (+ 30us) server_negotiation.cc:407] Connection header received
0928 02:57:04.910160 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.910165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.910371 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.910666 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:04.911455 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.913110 (+ 1655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.916092 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.917727 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.917921 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.918089 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.918746 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.918939 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10486,"thread_start_us":10315,"threads_started":1}
I20260928 02:57:04.919488 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.893571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34011 (local address 127.0.0.1:36806)
0928 02:57:04.902699 (+ 9128us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:04.902774 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:04.904039 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:04.912083 (+ 8044us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:04.912091 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:04.912131 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:04.912727 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.912740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.914682 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.914689 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.916509 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.916521 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.918561 (+ 2040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.918569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.918742 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.918784 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 02:57:04.918911 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:57:04.919055 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8725,"mutex_wait_us":1,"thread_start_us":15661,"threads_started":1}
I20260928 02:57:04.919891 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.893731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36806 (local address 127.27.229.193:34011)
0928 02:57:04.911461 (+ 17730us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.911469 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.911497 (+ 28us) server_negotiation.cc:407] Connection header received
0928 02:57:04.911595 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.911602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.911839 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.912042 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:04.912943 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.914548 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.916696 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.918251 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.918459 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.918614 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.919195 (+ 581us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.919360 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17546,"thread_start_us":17362,"threads_started":1}
I20260928 02:57:04.920011 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.893733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:38207 (local address 127.0.0.1:33692)
0928 02:57:04.909628 (+ 15895us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:04.909658 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:04.909816 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:04.910637 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:04.910644 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:04.910678 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:04.911188 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.911205 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.913266 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.913273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.915590 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.915606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.918146 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.918153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.918410 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.918482 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 02:57:04.918610 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:57:04.918855 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15712,"thread_start_us":18644,"threads_started":1}
I20260928 02:57:04.921100 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.894134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41073 (local address 127.0.0.1:44410)
0928 02:57:04.912731 (+ 18597us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:04.912751 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:04.912873 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:04.913730 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:04.913735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:04.913760 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:04.914189 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.914201 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.916058 (+ 1857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.916063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.918285 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:04.918301 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.920373 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.920382 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:04.920532 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.920554 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 02:57:04.920668 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:57:04.920794 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18452,"thread_start_us":8022,"threads_started":1}
I20260928 02:57:04.921200 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:04.894443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44410 (local address 127.27.229.194:41073)
0928 02:57:04.909689 (+ 15246us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:04.909700 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:04.912992 (+ 3292us) server_negotiation.cc:407] Connection header received
0928 02:57:04.913097 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:04.913105 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:04.913348 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:04.913554 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:04.914414 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.915872 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.918505 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:04.920034 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:04.920201 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:04.920363 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:04.920714 (+ 351us) server_negotiation.cc:299] Negotiation successful
0928 02:57:04.920877 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15040,"thread_start_us":15262,"threads_started":1}
I20260928 02:57:04.924166 13673 tablet_service.cc:1467] Processing CreateTablet for tablet a767ac6908e44389bff858fc670a8c8d (DEFAULT_TABLE table=test-table [id=da307ffdd33d4008b86441ecb1f71725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:04.924166 13405 tablet_service.cc:1467] Processing CreateTablet for tablet a767ac6908e44389bff858fc670a8c8d (DEFAULT_TABLE table=test-table [id=da307ffdd33d4008b86441ecb1f71725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:04.924477 13539 tablet_service.cc:1467] Processing CreateTablet for tablet a767ac6908e44389bff858fc670a8c8d (DEFAULT_TABLE table=test-table [id=da307ffdd33d4008b86441ecb1f71725]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:04.925702 13405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a767ac6908e44389bff858fc670a8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:04.925694 13673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a767ac6908e44389bff858fc670a8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:04.925700 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a767ac6908e44389bff858fc670a8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:04.939093 13757 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Bootstrap starting.
I20260928 02:57:04.940753 13758 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Bootstrap starting.
I20260928 02:57:04.942569 13759 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Bootstrap starting.
I20260928 02:57:04.942701 13757 tablet_bootstrap.cc:654] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:04.944070 13757 log.cc:824] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:04.945182 13758 tablet_bootstrap.cc:654] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:04.945700 13759 tablet_bootstrap.cc:654] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:04.946450 13758 log.cc:824] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:04.946878 13759 log.cc:824] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:04.947732 13757 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: No bootstrap required, opened a new log
I20260928 02:57:04.948071 13757 ts_tablet_manager.cc:1397] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:57:04.949429 13759 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: No bootstrap required, opened a new log
I20260928 02:57:04.949594 13758 tablet_bootstrap.cc:492] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: No bootstrap required, opened a new log
I20260928 02:57:04.949730 13759 ts_tablet_manager.cc:1397] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260928 02:57:04.949857 13758 ts_tablet_manager.cc:1397] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:57:04.958036 13759 raft_consensus.cc:348] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.958572 13759 raft_consensus.cc:374] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:04.958683 13759 raft_consensus.cc:729] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 447bd3b36cdb44c5a585fef881790602, State: Initialized, Role: FOLLOWER
I20260928 02:57:04.958992 13757 raft_consensus.cc:348] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.959434 13757 raft_consensus.cc:374] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:04.959535 13757 raft_consensus.cc:729] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9a4b8c8ef85441b91e5f67d5ec877e6, State: Initialized, Role: FOLLOWER
I20260928 02:57:04.959414 13759 consensus_queue.cc:260] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.960070 13757 consensus_queue.cc:260] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.960501 13758 raft_consensus.cc:348] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.960965 13758 raft_consensus.cc:374] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:04.961063 13758 raft_consensus.cc:729] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efbbf04d6049427ab62dc0d0f2b34c18, State: Initialized, Role: FOLLOWER
I20260928 02:57:04.961603 13604 heartbeater.cc:503] Master 127.27.229.254:42895 was elected leader, sending a full tablet report...
I20260928 02:57:04.961687 13758 consensus_queue.cc:260] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:04.962055 13757 ts_tablet_manager.cc:1428] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 02:57:04.963688 13758 ts_tablet_manager.cc:1428] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260928 02:57:04.965406 13738 heartbeater.cc:503] Master 127.27.229.254:42895 was elected leader, sending a full tablet report...
I20260928 02:57:04.966017 13759 ts_tablet_manager.cc:1428] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Time spent starting tablet: real 0.016s user 0.011s sys 0.006s
W20260928 02:57:04.993903 13471 tablet.cc:2369] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:05.033713 13605 tablet.cc:2369] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:05.070160 13739 tablet.cc:2369] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:05.294266 13764 raft_consensus.cc:482] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:05.294602 13764 raft_consensus.cc:504] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:05.296391 13764 leader_election.cc:288] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbbf04d6049427ab62dc0d0f2b34c18 (127.27.229.193:34011), 447bd3b36cdb44c5a585fef881790602 (127.27.229.195:38207)
I20260928 02:57:05.303444 13763 raft_consensus.cc:482] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:05.303807 13763 raft_consensus.cc:504] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:05.304476 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.296877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:35019 (local address 127.27.229.195:38207)
0928 02:57:05.297085 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.297091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.298293 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 02:57:05.298477 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.298484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.298576 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.298645 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:05.299321 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.300082 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.302195 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.303212 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.303406 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.303499 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.304029 (+ 530us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.304164 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:57:05.304519 13767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.297082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:38207 (local address 127.27.229.194:35019)
0928 02:57:05.298197 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:05.298211 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:05.298337 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:05.298700 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:05.298706 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:05.298736 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:05.299156 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.299167 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.300252 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.300258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.302008 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.302025 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.303643 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.303651 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.303844 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.303881 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 02:57:05.303997 (+ 116us) client_negotiation.cc:240] Negotiation successful
0928 02:57:05.304142 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":994,"thread_start_us":599,"threads_started":1}
I20260928 02:57:05.305120 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.296910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37245 (local address 127.27.229.193:34011)
0928 02:57:05.297099 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.297107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.298217 (+ 1110us) server_negotiation.cc:407] Connection header received
0928 02:57:05.298580 (+ 363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.298584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.298684 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.298775 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:05.299518 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.300437 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.302464 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.303557 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.303719 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.303791 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.304448 (+ 657us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.304878 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:57:05.305713 13766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.296932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34011 (local address 127.27.229.194:37245)
0928 02:57:05.298089 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:05.298107 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:05.298426 (+ 319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:05.298817 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:05.298820 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:05.298902 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:05.299307 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.299318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.300614 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.300619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.302155 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.302171 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.303931 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.303938 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.304234 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.304269 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:57:05.305157 (+ 888us) client_negotiation.cc:240] Negotiation successful
0928 02:57:05.305310 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":1,"spinlock_wait_cycles":23296,"thread_start_us":683,"threads_started":1}
I20260928 02:57:05.305567 13693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447bd3b36cdb44c5a585fef881790602" is_pre_election: true
I20260928 02:57:05.306084 13693 raft_consensus.cc:2454] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4b8c8ef85441b91e5f67d5ec877e6 in term 0.
I20260928 02:57:05.306879 13492 leader_election.cc:302] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447bd3b36cdb44c5a585fef881790602, f9a4b8c8ef85441b91e5f67d5ec877e6; no voters:
I20260928 02:57:05.306981 13763 leader_election.cc:288] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers efbbf04d6049427ab62dc0d0f2b34c18 (127.27.229.193:34011), f9a4b8c8ef85441b91e5f67d5ec877e6 (127.27.229.194:41073)
I20260928 02:57:05.307077 13425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" is_pre_election: true
I20260928 02:57:05.307679 13425 raft_consensus.cc:2454] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9a4b8c8ef85441b91e5f67d5ec877e6 in term 0.
I20260928 02:57:05.308076 13764 raft_consensus.cc:2781] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:05.308300 13764 raft_consensus.cc:482] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:05.308475 13764 raft_consensus.cc:3037] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:05.311766 13764 raft_consensus.cc:504] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:05.312876 13764 leader_election.cc:288] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [CANDIDATE]: Term 1 election: Requested vote from peers efbbf04d6049427ab62dc0d0f2b34c18 (127.27.229.193:34011), 447bd3b36cdb44c5a585fef881790602 (127.27.229.195:38207)
I20260928 02:57:05.312947 13425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbf04d6049427ab62dc0d0f2b34c18"
I20260928 02:57:05.312997 13693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "447bd3b36cdb44c5a585fef881790602"
I20260928 02:57:05.313179 13425 raft_consensus.cc:3037] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:05.313226 13693 raft_consensus.cc:3037] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:05.313613 13741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.306305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:34011 (local address 127.27.229.195:39619)
0928 02:57:05.306524 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:05.306544 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:05.306644 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:05.307041 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:05.307049 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:05.307080 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:05.307297 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.307309 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.308893 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.308902 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.311017 (+ 2115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.311032 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.312647 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.312653 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.312857 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.312892 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 02:57:05.312971 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 02:57:05.313115 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 02:57:05.314149 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.306483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:39619 (local address 127.27.229.193:34011)
0928 02:57:05.306677 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.306686 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.306704 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:57:05.306770 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.306777 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.306880 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.307455 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:05.307593 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.308758 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.311191 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.312449 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.313383 (+ 934us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.313460 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.313524 (+ 64us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.313680 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:57:05.314869 13424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "447bd3b36cdb44c5a585fef881790602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" is_pre_election: true
I20260928 02:57:05.316628 13693 raft_consensus.cc:2454] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4b8c8ef85441b91e5f67d5ec877e6 in term 1.
I20260928 02:57:05.316807 13425 raft_consensus.cc:2454] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9a4b8c8ef85441b91e5f67d5ec877e6 in term 1.
I20260928 02:57:05.317057 13492 leader_election.cc:302] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 447bd3b36cdb44c5a585fef881790602, f9a4b8c8ef85441b91e5f67d5ec877e6; no voters:
I20260928 02:57:05.317449 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.307653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:51947 (local address 127.27.229.194:41073)
0928 02:57:05.307888 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.307896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.308674 (+ 778us) server_negotiation.cc:407] Connection header received
0928 02:57:05.308933 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.308937 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.308991 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.309034 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:05.309719 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.310467 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.314861 (+ 4394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.316181 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.316322 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.316392 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.317043 (+ 651us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.317190 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:57:05.317561 13764 raft_consensus.cc:2781] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:05.318030 13768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.307496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:41073 (local address 127.27.229.195:51947)
0928 02:57:05.308584 (+ 1088us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:05.308602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:05.308768 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:05.309179 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:05.309185 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:05.309204 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:05.309534 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.309544 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.310624 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.310630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.312705 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:05.312717 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.316628 (+ 3911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.316637 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:05.316823 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.316866 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:57:05.317632 (+ 766us) client_negotiation.cc:240] Negotiation successful
0928 02:57:05.317779 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"thread_start_us":1059,"threads_started":1}
I20260928 02:57:05.318080 13764 raft_consensus.cc:686] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 1 LEADER]: Becoming Leader. State: Replica: f9a4b8c8ef85441b91e5f67d5ec877e6, State: Running, Role: LEADER
I20260928 02:57:05.318969 13764 consensus_queue.cc:237] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } }
I20260928 02:57:05.319075 13559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a767ac6908e44389bff858fc670a8c8d" candidate_uuid: "447bd3b36cdb44c5a585fef881790602" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" is_pre_election: true
I20260928 02:57:05.320144 13627 leader_election.cc:302] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 447bd3b36cdb44c5a585fef881790602; no voters: efbbf04d6049427ab62dc0d0f2b34c18, f9a4b8c8ef85441b91e5f67d5ec877e6
I20260928 02:57:05.320608 13763 raft_consensus.cc:2726] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 02:57:05.324191 13280 catalog_manager.cc:5495] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f9a4b8c8ef85441b91e5f67d5ec877e6 (127.27.229.194). New cstate: current_term: 1 leader_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9a4b8c8ef85441b91e5f67d5ec877e6" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 41073 } health_report { overall_health: HEALTHY } } }
I20260928 02:57:05.386308 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.380097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44420 (local address 127.27.229.194:41073)
0928 02:57:05.380330 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.380339 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.380624 (+ 285us) server_negotiation.cc:407] Connection header received
0928 02:57:05.380887 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.380892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.380981 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.381140 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:05.381597 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.382383 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.383980 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.384405 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.384546 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.385100 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:05.385639 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:05.385821 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.385873 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.385971 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:57:05.395347 13425 raft_consensus.cc:1261] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 1 FOLLOWER]: Refusing update from remote peer f9a4b8c8ef85441b91e5f67d5ec877e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:05.395375 13693 raft_consensus.cc:1261] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 1 FOLLOWER]: Refusing update from remote peer f9a4b8c8ef85441b91e5f67d5ec877e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:05.396327 13769 consensus_queue.cc:1035] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "447bd3b36cdb44c5a585fef881790602" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 38207 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:05.396725 13764 consensus_queue.cc:1035] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efbbf04d6049427ab62dc0d0f2b34c18" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:05.415620 13773 mvcc.cc:205] Tried to move back new op lower bound from 7334151067208196096 to 7334151066911035392. Current Snapshot: MvccSnapshot[applied={T|T < 7334151067208196096 or (T in {7334151067208196096})}]
I20260928 02:57:05.416294 13775 mvcc.cc:205] Tried to move back new op lower bound from 7334151067208196096 to 7334151066911035392. Current Snapshot: MvccSnapshot[applied={T|T < 7334151067208196096}]
I20260928 02:57:05.417140 13774 mvcc.cc:205] Tried to move back new op lower bound from 7334151067208196096 to 7334151066911035392. Current Snapshot: MvccSnapshot[applied={T|T < 7334151067208196096}]
I20260928 02:57:05.424420 13280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:05.424708 13280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:05.428375 13280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88715cc2f69c431c8f43f82e30200e8c: Sending DeleteTablet for 3 replicas of tablet a767ac6908e44389bff858fc670a8c8d
I20260928 02:57:05.429687 13673 tablet_service.cc:1514] Processing DeleteTablet for tablet a767ac6908e44389bff858fc670a8c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:05 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:33692
I20260928 02:57:05.429687 13405 tablet_service.cc:1514] Processing DeleteTablet for tablet a767ac6908e44389bff858fc670a8c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:05 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:36806
I20260928 02:57:05.429726 13539 tablet_service.cc:1514] Processing DeleteTablet for tablet a767ac6908e44389bff858fc670a8c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:05 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:44410
I20260928 02:57:05.432101 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13342
I20260928 02:57:05.442001 13783 tablet_replica.cc:330] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: stopping tablet replica
I20260928 02:57:05.442309 13785 tablet_replica.cc:330] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: stopping tablet replica
I20260928 02:57:05.442524 13783 raft_consensus.cc:2229] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:05.442878 13785 raft_consensus.cc:2229] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:05.443902 13785 raft_consensus.cc:2258] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:05.444859 13783 raft_consensus.cc:2258] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:05.446489 13785 ts_tablet_manager.cc:1905] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:05.447647 13783 ts_tablet_manager.cc:1905] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:05.452287 13784 tablet_replica.cc:330] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: stopping tablet replica
Sep 28 02:57:05 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:57:05.452771 13784 raft_consensus.cc:2229] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:05.453327 13784 raft_consensus.cc:2258] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:05.454927 13785 ts_tablet_manager.cc:1918] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:05.455160 13785 log.cc:1196] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/a767ac6908e44389bff858fc670a8c8d
I20260928 02:57:05.455739 13784 ts_tablet_manager.cc:1905] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:05.455901 13785 ts_tablet_manager.cc:1939] T a767ac6908e44389bff858fc670a8c8d P f9a4b8c8ef85441b91e5f67d5ec877e6: Deleting consensus metadata
I20260928 02:57:05.457830 13266 catalog_manager.cc:4841] TS f9a4b8c8ef85441b91e5f67d5ec877e6 (127.27.229.194:41073): tablet a767ac6908e44389bff858fc670a8c8d (table test-table [id=da307ffdd33d4008b86441ecb1f71725]) successfully deleted
I20260928 02:57:05.461127 13783 ts_tablet_manager.cc:1918] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:05.461328 13783 log.cc:1196] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/a767ac6908e44389bff858fc670a8c8d
I20260928 02:57:05.461894 13783 ts_tablet_manager.cc:1939] T a767ac6908e44389bff858fc670a8c8d P efbbf04d6049427ab62dc0d0f2b34c18: Deleting consensus metadata
I20260928 02:57:05.462543 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.433569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.27.229.193:34011)
0928 02:57:05.433749 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.433755 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.438452 (+ 4697us) server_negotiation.cc:407] Connection header received
0928 02:57:05.438682 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.438687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.438782 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.438921 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:05.443741 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.444904 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.446897 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.447615 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.447724 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.453794 (+ 6070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:05.453814 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:05.453832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:05.453882 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:05.458868 (+ 4986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.459844 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.459854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.459858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.460037 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.460504 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.460509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.460513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.460994 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:05.461357 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.461862 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.462134 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:57:05.463263 13465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:05.469430 13784 ts_tablet_manager.cc:1918] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:05.469655 13784 log.cc:1196] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/a767ac6908e44389bff858fc670a8c8d
I20260928 02:57:05.470101 13784 ts_tablet_manager.cc:1939] T a767ac6908e44389bff858fc670a8c8d P 447bd3b36cdb44c5a585fef881790602: Deleting consensus metadata
I20260928 02:57:05.472023 13265 catalog_manager.cc:4841] TS 447bd3b36cdb44c5a585fef881790602 (127.27.229.195:38207): tablet a767ac6908e44389bff858fc670a8c8d (table test-table [id=da307ffdd33d4008b86441ecb1f71725]) successfully deleted
I20260928 02:57:05.560887 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13342
I20260928 02:57:05.561079 13267 catalog_manager.cc:4841] TS efbbf04d6049427ab62dc0d0f2b34c18 (127.27.229.193:34011): tablet a767ac6908e44389bff858fc670a8c8d (table test-table [id=da307ffdd33d4008b86441ecb1f71725]) successfully deleted
I20260928 02:57:05.586087 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13476
Sep 28 02:57:05 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:57:05.603502 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.586927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.27.229.194:41073)
0928 02:57:05.587099 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.587104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.587270 (+ 166us) server_negotiation.cc:407] Connection header received
0928 02:57:05.587500 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.587509 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.587594 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.587702 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:05.588991 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.589764 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.591228 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.591755 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.591895 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.596722 (+ 4827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:05.596738 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:05.596754 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:05.596795 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:05.600365 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.601152 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.601159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.601161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.601312 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.601724 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.601728 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.601730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.602208 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:05.602530 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.602904 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.603149 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:57:05.604173 13599 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:05.721728 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13476
I20260928 02:57:05.740777 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13610
Sep 28 02:57:05 dist-test-slave-s1kg krb5kdc[13236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:57:05.757100 13756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:05.741683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33706 (local address 127.27.229.195:38207)
0928 02:57:05.741861 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:05.741865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:05.741946 (+ 81us) server_negotiation.cc:407] Connection header received
0928 02:57:05.742064 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:05.742069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:05.742143 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:05.742227 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:05.743623 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.744434 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.745756 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:05.746170 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:05.746291 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:05.751047 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:05.751062 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:05.751078 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:05.751116 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:05.754080 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.754931 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.754945 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.754951 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.755137 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:05.755592 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:05.755596 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:05.755598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:05.755994 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:05.756219 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:05.756539 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 02:57:05.756713 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:57:05.757869 13733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:05.850700 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13610
I20260928 02:57:05.868944 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13249
I20260928 02:57:05.870229 13311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:05.982509 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13249
2026-09-28T02:57:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T02:57:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:06Z Disabled control of system clock
I20260928 02:57:06.061908 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44673
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:06.399506 13794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:06.399816 13794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:06.399899 13794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:06.407670 13794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:06.407806 13794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:06.407846 13794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:06.407896 13794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:06.407943 13794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:06.417487 13794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:44673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:06.418798 13794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:06.420528 13794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:06.433315 13800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:06.433400 13799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:06.433353 13802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:06.434636 13794 server_base.cc:1034] running on GCE node
I20260928 02:57:06.435504 13794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:06.437036 13794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:06.438273 13794 hybrid_clock.cc:648] HybridClock initialized: now 1790564226438234 us; error 64 us; skew 500 ppm
I20260928 02:57:06.438673 13794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:06.441147 13794 webserver.cc:460] Webserver started at http://127.27.229.254:34065/ using document root <none> and password file <none>
I20260928 02:57:06.441859 13794 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:06.441998 13794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:06.442325 13794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:06.445459 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "483b206fd6524dabb52d3c5431e05181"
format_stamp: "Formatted at 2026-09-28 02:57:06 on dist-test-slave-s1kg"
server_key: "af981be2456bc492ced8be30d08bfac5"
server_key_iv: "cfe67dbfb68f9adf8a309226e33f8b21"
server_key_version: "encryptionkey@0"
I20260928 02:57:06.446300 13794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "483b206fd6524dabb52d3c5431e05181"
format_stamp: "Formatted at 2026-09-28 02:57:06 on dist-test-slave-s1kg"
server_key: "af981be2456bc492ced8be30d08bfac5"
server_key_iv: "cfe67dbfb68f9adf8a309226e33f8b21"
server_key_version: "encryptionkey@0"
I20260928 02:57:06.451684 13794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260928 02:57:06.455410 13808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:06.456846 13794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:57:06.457049 13794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "483b206fd6524dabb52d3c5431e05181"
format_stamp: "Formatted at 2026-09-28 02:57:06 on dist-test-slave-s1kg"
server_key: "af981be2456bc492ced8be30d08bfac5"
server_key_iv: "cfe67dbfb68f9adf8a309226e33f8b21"
server_key_version: "encryptionkey@0"
I20260928 02:57:06.457288 13794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:06.484712 13794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:06.485662 13794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:06.485980 13794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:06.505165 13794 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:44673
I20260928 02:57:06.505190 13859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:44673 every 8 connection(s)
I20260928 02:57:06.506618 13794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:06.510561 13860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:06.516909 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13794
I20260928 02:57:06.517119 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:06.517585 28567 external_mini_cluster.cc:1428] Setting key 85b231c86f41eeb8e4f2941afaa1d0ef
I20260928 02:57:06.522526 13860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: Bootstrap starting.
I20260928 02:57:06.526803 13860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:06.527987 13860 log.cc:824] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:06.531698 13860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: No bootstrap required, opened a new log
I20260928 02:57:06.534174 13863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:06.519699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.27.229.254:44673)
0928 02:57:06.520804 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:06.520819 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:06.520907 (+ 88us) server_negotiation.cc:407] Connection header received
0928 02:57:06.523470 (+ 2563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:06.523499 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:06.524098 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:06.524634 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:06.525251 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:06.527277 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:06.528978 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:06.529858 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:06.530047 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:06.531214 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:06.531224 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:06.531244 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:06.531987 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:06.532091 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:06.532120 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:06.532267 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:06.532512 (+ 245us) server_negotiation.cc:299] Negotiation successful
0928 02:57:06.532846 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":322,"threads_started":1}
I20260928 02:57:06.540666 13860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } }
I20260928 02:57:06.541024 13860 raft_consensus.cc:374] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:06.541102 13860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 483b206fd6524dabb52d3c5431e05181, State: Initialized, Role: FOLLOWER
I20260928 02:57:06.541657 13860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } }
I20260928 02:57:06.541860 13860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:06.541960 13860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:06.542088 13860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:06.544603 13860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } }
I20260928 02:57:06.545056 13860 leader_election.cc:302] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 483b206fd6524dabb52d3c5431e05181; no voters:
I20260928 02:57:06.545641 13860 leader_election.cc:288] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:06.545900 13865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:06.546655 13865 raft_consensus.cc:686] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [term 1 LEADER]: Becoming Leader. State: Replica: 483b206fd6524dabb52d3c5431e05181, State: Running, Role: LEADER
I20260928 02:57:06.547204 13865 consensus_queue.cc:237] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } }
I20260928 02:57:06.547801 13860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:06.550622 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "483b206fd6524dabb52d3c5431e05181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } } }
I20260928 02:57:06.551137 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:06.551074 13867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 483b206fd6524dabb52d3c5431e05181. Latest consensus state: current_term: 1 leader_uuid: "483b206fd6524dabb52d3c5431e05181" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "483b206fd6524dabb52d3c5431e05181" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44673 } } }
I20260928 02:57:06.551395 13867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:06.552881 13874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:06.558430 13874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:06.567467 13874 catalog_manager.cc:1347] Generated new cluster ID: 1143e1809d54405298dd8b7525de707f
I20260928 02:57:06.567590 13874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:06.603278 13874 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:06.604661 13874 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:06.619187 13874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: Generated new TSK 0
I20260928 02:57:06.620086 13874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:06.625233 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--builtin_ntp_servers=127.27.229.212:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:06.976467 13884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:06.976817 13884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:06.976985 13884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:06.984567 13884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:06.984722 13884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:06.984936 13884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:06.994469 13884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:06.995846 13884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:06.997926 13884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:07.012163 13890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.012128 13889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.012190 13892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:07.013016 13884 server_base.cc:1034] running on GCE node
I20260928 02:57:07.013665 13884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:07.014725 13884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:07.016010 13884 hybrid_clock.cc:648] HybridClock initialized: now 1790564227015917 us; error 92 us; skew 500 ppm
I20260928 02:57:07.016449 13884 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:07.019142 13884 webserver.cc:460] Webserver started at http://127.27.229.193:40481/ using document root <none> and password file <none>
I20260928 02:57:07.019917 13884 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:07.020020 13884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:07.020398 13884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:07.023504 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "ed81f01909134a5e9017894fcf342a30"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "00eac7848babc90d9703f57c42008826"
server_key_iv: "ba761cba72270e25f8fce7b288bb0902"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.024374 13884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "ed81f01909134a5e9017894fcf342a30"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "00eac7848babc90d9703f57c42008826"
server_key_iv: "ba761cba72270e25f8fce7b288bb0902"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.030078 13884 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:57:07.033804 13898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.035446 13884 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:07.035633 13884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "ed81f01909134a5e9017894fcf342a30"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "00eac7848babc90d9703f57c42008826"
server_key_iv: "ba761cba72270e25f8fce7b288bb0902"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.035843 13884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:07.070032 13884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:07.070994 13884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:07.071305 13884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:07.072341 13884 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:07.073884 13884 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:07.073978 13884 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.074134 13884 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:07.074239 13884 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.109651 13884 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:40427
I20260928 02:57:07.109751 14010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:40427 every 8 connection(s)
I20260928 02:57:07.111061 13884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:07.117005 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 13884
I20260928 02:57:07.117206 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:07.117656 28567 external_mini_cluster.cc:1428] Setting key 2ac0edaea181e327bd29df56682aa20c
I20260928 02:57:07.120609 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--builtin_ntp_servers=127.27.229.212:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:07.124096 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:07.113518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:43563 (local address 127.27.229.254:44673)
0928 02:57:07.113974 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:07.113978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:07.115039 (+ 1061us) server_negotiation.cc:407] Connection header received
0928 02:57:07.116895 (+ 1856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:07.116900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:07.116996 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:07.117123 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:07.118303 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.119305 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.121137 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.121633 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.121773 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:07.123251 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:07.123256 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:07.123259 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:07.123278 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:07.123315 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:07.123328 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:07.123432 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:07.123650 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:57:07.123749 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":247,"threads_started":1}
I20260928 02:57:07.125136 14014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:07.113958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:44673 (local address 127.27.229.193:43563)
0928 02:57:07.114803 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:07.114909 (+ 106us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:07.116586 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:07.117392 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:07.117401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:07.117491 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:07.118120 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:07.118139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.119548 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.119553 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:07.120986 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:07.120999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.121876 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.121886 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:07.122125 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:07.122898 (+ 773us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:07.122902 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:07.122960 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:07.123043 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:07.123426 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:07.123438 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:07.123448 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:57:07.123640 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 02:57:07.124052 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":255,"threads_started":1}
I20260928 02:57:07.126675 14011 heartbeater.cc:347] Connected to a master server at 127.27.229.254:44673
I20260928 02:57:07.127022 14011 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:07.127767 14011 heartbeater.cc:511] Master 127.27.229.254:44673 requested a full tablet report, sending...
I20260928 02:57:07.130322 13825 ts_manager.cc:194] Registered new tserver with Master: ed81f01909134a5e9017894fcf342a30 (127.27.229.193:40427)
I20260928 02:57:07.132496 13825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.193:43563
W20260928 02:57:07.457116 14015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:07.457422 14015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:07.457513 14015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:07.466056 14015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:07.466163 14015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:07.466285 14015 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:07.475775 14015 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:07.477128 14015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:07.478861 14015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:07.492717 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.492676 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.492762 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:07.493741 14015 server_base.cc:1034] running on GCE node
I20260928 02:57:07.494387 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:07.495464 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:07.496681 14015 hybrid_clock.cc:648] HybridClock initialized: now 1790564227496638 us; error 32 us; skew 500 ppm
I20260928 02:57:07.496980 14015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:07.499289 14015 webserver.cc:460] Webserver started at http://127.27.229.194:34719/ using document root <none> and password file <none>
I20260928 02:57:07.499944 14015 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:07.500030 14015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:07.500303 14015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:07.503273 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "a9a40171df67487eb48c41737d5f5ec7"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "d3c6d60e3ce3b8194b426a53aa73f0b0"
server_key_iv: "575aef065c23007f661bde7ffa608efc"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.504122 14015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "a9a40171df67487eb48c41737d5f5ec7"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "d3c6d60e3ce3b8194b426a53aa73f0b0"
server_key_iv: "575aef065c23007f661bde7ffa608efc"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.510979 14015 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 02:57:07.515456 14029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.517088 14015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:07.517267 14015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "a9a40171df67487eb48c41737d5f5ec7"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "d3c6d60e3ce3b8194b426a53aa73f0b0"
server_key_iv: "575aef065c23007f661bde7ffa608efc"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.517551 14015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:07.542906 14015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:07.543738 14015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:07.544028 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:07.545028 14015 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:07.546476 14015 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:07.546563 14015 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.546635 14015 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:07.546692 14015 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.582553 14015 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:40891
I20260928 02:57:07.582693 14141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:40891 every 8 connection(s)
I20260928 02:57:07.584009 14015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:07.588405 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14015
I20260928 02:57:07.588591 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:07.589165 28567 external_mini_cluster.cc:1428] Setting key f9ecfc2416c99233616840798059da9a
I20260928 02:57:07.597031 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--builtin_ntp_servers=127.27.229.212:38011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:07.598809 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:07.586690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:58607 (local address 127.27.229.254:44673)
0928 02:57:07.586942 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:07.586948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:07.588333 (+ 1385us) server_negotiation.cc:407] Connection header received
0928 02:57:07.591165 (+ 2832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:07.591169 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:07.591252 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:07.591344 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:07.592980 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.593772 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.595945 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.596367 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.596516 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:07.597984 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:07.597989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:07.597992 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:07.598010 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:07.598040 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:07.598051 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:07.598149 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:07.598395 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 02:57:07.598497 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 02:57:07.599881 14144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:07.587170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:44673 (local address 127.27.229.194:58607)
0928 02:57:07.588124 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:07.588171 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:07.590749 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:07.591656 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:07.591672 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:07.591820 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:07.592728 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:07.592757 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.593960 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.593965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:07.595773 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:07.595789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:07.596613 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:07.596621 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:07.596862 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:07.597627 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:07.597630 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:07.597673 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:07.597762 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:07.598148 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:07.598162 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:07.598172 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:57:07.598382 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 02:57:07.598805 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260928 02:57:07.601336 14142 heartbeater.cc:347] Connected to a master server at 127.27.229.254:44673
I20260928 02:57:07.601681 14142 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:07.602660 14142 heartbeater.cc:511] Master 127.27.229.254:44673 requested a full tablet report, sending...
I20260928 02:57:07.604629 13825 ts_manager.cc:194] Registered new tserver with Master: a9a40171df67487eb48c41737d5f5ec7 (127.27.229.194:40891)
I20260928 02:57:07.605509 13825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.194:58607
W20260928 02:57:07.929836 14145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:07.930145 14145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:07.930307 14145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:07.937999 14145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:07.938145 14145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:07.938340 14145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:07.948706 14145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:07.950076 14145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:07.952018 14145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:07.966202 14151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.966243 14150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:07.966202 14153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:07.967357 14145 server_base.cc:1034] running on GCE node
I20260928 02:57:07.968067 14145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:07.969125 14145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:07.970383 14145 hybrid_clock.cc:648] HybridClock initialized: now 1790564227970345 us; error 37 us; skew 500 ppm
I20260928 02:57:07.970754 14145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:07.973618 14145 webserver.cc:460] Webserver started at http://127.27.229.195:43435/ using document root <none> and password file <none>
I20260928 02:57:07.974399 14145 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:07.974556 14145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:07.975035 14145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:07.978154 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "aae31ac2fd8f437094da0fec2001e7ed"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "395a21b2afd3cc0a409aba251c5ce339"
server_key_iv: "5c8658dde47af203339eabaa5284f4c9"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.979094 14145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "aae31ac2fd8f437094da0fec2001e7ed"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "395a21b2afd3cc0a409aba251c5ce339"
server_key_iv: "5c8658dde47af203339eabaa5284f4c9"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.984752 14145 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:57:07.988564 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:07.989925 14145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:57:07.990135 14145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "aae31ac2fd8f437094da0fec2001e7ed"
format_stamp: "Formatted at 2026-09-28 02:57:07 on dist-test-slave-s1kg"
server_key: "395a21b2afd3cc0a409aba251c5ce339"
server_key_iv: "5c8658dde47af203339eabaa5284f4c9"
server_key_version: "encryptionkey@0"
I20260928 02:57:07.990366 14145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:08.016956 14145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:08.017891 14145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:08.018213 14145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:08.019325 14145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:08.020952 14145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:08.021078 14145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:08.021206 14145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:08.021296 14145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:08.057195 14145 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:34469
I20260928 02:57:08.057301 14271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:34469 every 8 connection(s)
I20260928 02:57:08.058615 14145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:08.063971 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14145
I20260928 02:57:08.064167 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:08.064642 28567 external_mini_cluster.cc:1428] Setting key 13700b9885f9e6206ab0900f3676c913
I20260928 02:57:08.071897 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.061529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:56137 (local address 127.27.229.254:44673)
0928 02:57:08.061731 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.061735 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.063197 (+ 1462us) server_negotiation.cc:407] Connection header received
0928 02:57:08.065159 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.065163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.065234 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.065323 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:08.066554 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.067519 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.069116 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.069543 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.069650 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.071129 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:08.071134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:08.071137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:08.071156 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:08.071188 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:08.071200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:08.071301 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.071561 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.071643 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:57:08.073557 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.062001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:44673 (local address 127.27.229.195:56137)
0928 02:57:08.063032 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.063069 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.064878 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.065552 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.065561 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.065657 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:08.066373 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.066400 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.067695 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.067700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.068961 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.068973 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.069786 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.069791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.069997 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.070714 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:08.070717 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:08.070757 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:08.070924 (+ 167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:08.071355 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:08.071364 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:08.071371 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.071534 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.071863 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":324,"threads_started":1}
I20260928 02:57:08.075277 14272 heartbeater.cc:347] Connected to a master server at 127.27.229.254:44673
I20260928 02:57:08.075634 14272 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:08.076467 14272 heartbeater.cc:511] Master 127.27.229.254:44673 requested a full tablet report, sending...
I20260928 02:57:08.078068 13825 ts_manager.cc:194] Registered new tserver with Master: aae31ac2fd8f437094da0fec2001e7ed (127.27.229.195:34469)
I20260928 02:57:08.079180 13825 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.195:56137
I20260928 02:57:08.080724 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:08.091480 14013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.085004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.27.229.254:44673)
0928 02:57:08.085192 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.085196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.085675 (+ 479us) server_negotiation.cc:407] Connection header received
0928 02:57:08.085866 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.085871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.085957 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.086052 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:08.086564 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.087485 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.089142 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.089593 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.089695 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.090798 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:08.090802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:08.090806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:08.090877 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:08.090914 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:08.090926 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:08.090999 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.091138 (+ 139us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.091217 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:57:08.095830 13825 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45976:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:08.103576 13825 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:08.135788 14011 heartbeater.cc:503] Master 127.27.229.254:44673 was elected leader, sending a full tablet report...
I20260928 02:57:08.152889 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.117192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40891 (local address 127.0.0.1:35798)
0928 02:57:08.143740 (+ 26548us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.143759 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.143907 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.145009 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.145015 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.145051 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:08.145784 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.145798 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147800 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.147806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.150215 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.150229 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.152015 (+ 1786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.152021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.152175 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.152211 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.152326 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.152481 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26403,"mutex_wait_us":56,"spinlock_wait_cycles":896,"thread_start_us":27208,"threads_started":1}
I20260928 02:57:08.152951 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.117824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35798 (local address 127.27.229.194:40891)
0928 02:57:08.142392 (+ 24568us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.142405 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.144052 (+ 1647us) server_negotiation.cc:407] Connection header received
0928 02:57:08.144162 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.144169 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.144568 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.144767 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:08.145992 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147601 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.150484 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.151705 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.151885 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.152259 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.152391 (+ 132us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.152594 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24353,"thread_start_us":24254,"threads_started":1}
I20260928 02:57:08.154495 14287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.117154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:37754)
0928 02:57:08.143260 (+ 26106us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.143328 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.143753 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.144989 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.144996 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.145045 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:08.145610 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.145631 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147770 (+ 2139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.147775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.151379 (+ 3604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.151394 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.153641 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.153650 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.153828 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.153862 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.153951 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.154085 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25714,"mutex_wait_us":802,"spinlock_wait_cycles":3456,"thread_start_us":26176,"threads_started":1}
I20260928 02:57:08.155382 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.117194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37754 (local address 127.27.229.193:40427)
0928 02:57:08.142949 (+ 25755us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.142962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.143508 (+ 546us) server_negotiation.cc:407] Connection header received
0928 02:57:08.144107 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.144120 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.144484 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.144656 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:08.145870 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147583 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.151644 (+ 4061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.153419 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.153627 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.153821 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.154868 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.155056 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25481,"thread_start_us":25774,"threads_started":1}
I20260928 02:57:08.156189 14286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.117572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:34469 (local address 127.0.0.1:59398)
0928 02:57:08.144685 (+ 27113us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.144705 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.144812 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.145777 (+ 965us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.145783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.145815 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:08.146130 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.146138 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147827 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.147832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.151378 (+ 3546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.151394 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.155425 (+ 4031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.155434 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.155615 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.155649 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.155750 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.155904 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26980,"mutex_wait_us":573,"thread_start_us":26508,"threads_started":1}
I20260928 02:57:08.156303 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.118360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.27.229.195:34469)
0928 02:57:08.141586 (+ 23226us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.141613 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.144964 (+ 3351us) server_negotiation.cc:407] Connection header received
0928 02:57:08.145082 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.145091 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.145322 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.145522 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:08.146323 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.147558 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.153932 (+ 6374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.155203 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.155429 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.155646 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.155798 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.155955 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22942,"thread_start_us":23263,"threads_started":1}
I20260928 02:57:08.157516 14077 tablet_service.cc:1467] Processing CreateTablet for tablet 708200b16300405f9c9357c40351b257 (DEFAULT_TABLE table=test-table [id=7f3cf3a079694c45853ef227206f5d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:08.158795 14077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708200b16300405f9c9357c40351b257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:08.159729 14207 tablet_service.cc:1467] Processing CreateTablet for tablet 708200b16300405f9c9357c40351b257 (DEFAULT_TABLE table=test-table [id=7f3cf3a079694c45853ef227206f5d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:08.161398 14207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708200b16300405f9c9357c40351b257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:08.162717 13946 tablet_service.cc:1467] Processing CreateTablet for tablet 708200b16300405f9c9357c40351b257 (DEFAULT_TABLE table=test-table [id=7f3cf3a079694c45853ef227206f5d30]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:08.164237 13946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708200b16300405f9c9357c40351b257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:08.172154 14290 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Bootstrap starting.
I20260928 02:57:08.176170 14291 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Bootstrap starting.
I20260928 02:57:08.176535 14290 tablet_bootstrap.cc:654] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:08.177934 14290 log.cc:824] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:08.178654 14292 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Bootstrap starting.
I20260928 02:57:08.180117 14291 tablet_bootstrap.cc:654] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:08.181528 14292 tablet_bootstrap.cc:654] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:08.181861 14291 log.cc:824] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:08.182547 14292 log.cc:824] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:08.186105 14290 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: No bootstrap required, opened a new log
I20260928 02:57:08.186128 14292 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: No bootstrap required, opened a new log
I20260928 02:57:08.186128 14291 tablet_bootstrap.cc:492] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: No bootstrap required, opened a new log
I20260928 02:57:08.186399 14290 ts_tablet_manager.cc:1397] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260928 02:57:08.186412 14292 ts_tablet_manager.cc:1397] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260928 02:57:08.186426 14291 ts_tablet_manager.cc:1397] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:57:08.193503 14290 raft_consensus.cc:348] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.193964 14290 raft_consensus.cc:374] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:08.194087 14290 raft_consensus.cc:729] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a40171df67487eb48c41737d5f5ec7, State: Initialized, Role: FOLLOWER
I20260928 02:57:08.194911 14290 consensus_queue.cc:260] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.196614 14142 heartbeater.cc:503] Master 127.27.229.254:44673 was elected leader, sending a full tablet report...
I20260928 02:57:08.197359 14290 ts_tablet_manager.cc:1428] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260928 02:57:08.197186 14292 raft_consensus.cc:348] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.197186 14291 raft_consensus.cc:348] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.197618 14291 raft_consensus.cc:374] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:08.197614 14292 raft_consensus.cc:374] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:08.197708 14291 raft_consensus.cc:729] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aae31ac2fd8f437094da0fec2001e7ed, State: Initialized, Role: FOLLOWER
I20260928 02:57:08.197708 14292 raft_consensus.cc:729] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed81f01909134a5e9017894fcf342a30, State: Initialized, Role: FOLLOWER
I20260928 02:57:08.198410 14292 consensus_queue.cc:260] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.198421 14291 consensus_queue.cc:260] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.200243 14291 ts_tablet_manager.cc:1428] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260928 02:57:08.200537 14292 ts_tablet_manager.cc:1428] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
I20260928 02:57:08.200668 14272 heartbeater.cc:503] Master 127.27.229.254:44673 was elected leader, sending a full tablet report...
W20260928 02:57:08.310565 14273 tablet.cc:2369] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:08.336174 14143 tablet.cc:2369] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:08.363492 14012 tablet.cc:2369] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:08.445194 14298 raft_consensus.cc:482] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:08.445556 14298 raft_consensus.cc:504] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.447316 14298 leader_election.cc:288] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed81f01909134a5e9017894fcf342a30 (127.27.229.193:40427), a9a40171df67487eb48c41737d5f5ec7 (127.27.229.194:40891)
I20260928 02:57:08.453269 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.447942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:44725 (local address 127.27.229.194:40891)
0928 02:57:08.448132 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.448138 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.448186 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:57:08.448265 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.448269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.448340 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.448488 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:08.448896 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.449682 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.451608 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.452339 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.452523 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.452597 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.452869 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.452996 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 02:57:08.453287 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.447589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:40891 (local address 127.27.229.195:44725)
0928 02:57:08.447792 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.447814 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.447949 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.448509 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.448514 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.448543 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:08.448741 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.448750 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.449814 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.449821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.451420 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.451433 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.452508 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.452514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.452706 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.452730 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.452840 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.452993 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:57:08.453841 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.447843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:44449 (local address 127.27.229.193:40427)
0928 02:57:08.448073 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.448080 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.448780 (+ 700us) server_negotiation.cc:407] Connection header received
0928 02:57:08.448978 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.448982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.449051 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.449152 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:08.449661 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.450350 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.452157 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.452960 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.453127 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.453191 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.453536 (+ 345us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.453654 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:57:08.454515 14299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.447603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.27.229.195:44449)
0928 02:57:08.448670 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.448688 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:08.448833 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:08.449186 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:08.449190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:08.449212 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:08.449514 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.449522 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.450516 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.450521 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.451988 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:08.451999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.453132 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.453138 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:08.453313 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.453362 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 02:57:08.454035 (+ 673us) client_negotiation.cc:240] Negotiation successful
0928 02:57:08.454170 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"mutex_wait_us":128,"thread_start_us":561,"threads_started":1}
I20260928 02:57:08.454452 14097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708200b16300405f9c9357c40351b257" candidate_uuid: "aae31ac2fd8f437094da0fec2001e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a40171df67487eb48c41737d5f5ec7" is_pre_election: true
I20260928 02:57:08.455067 14097 raft_consensus.cc:2454] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae31ac2fd8f437094da0fec2001e7ed in term 0.
I20260928 02:57:08.455621 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708200b16300405f9c9357c40351b257" candidate_uuid: "aae31ac2fd8f437094da0fec2001e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed81f01909134a5e9017894fcf342a30" is_pre_election: true
I20260928 02:57:08.455761 14160 leader_election.cc:302] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9a40171df67487eb48c41737d5f5ec7, aae31ac2fd8f437094da0fec2001e7ed; no voters:
I20260928 02:57:08.456068 13966 raft_consensus.cc:2454] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aae31ac2fd8f437094da0fec2001e7ed in term 0.
I20260928 02:57:08.456200 14298 raft_consensus.cc:2781] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:08.456344 14298 raft_consensus.cc:482] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:08.456452 14298 raft_consensus.cc:3037] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:08.458725 14298 raft_consensus.cc:504] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.459455 14298 leader_election.cc:288] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [CANDIDATE]: Term 1 election: Requested vote from peers ed81f01909134a5e9017894fcf342a30 (127.27.229.193:40427), a9a40171df67487eb48c41737d5f5ec7 (127.27.229.194:40891)
I20260928 02:57:08.459849 13966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708200b16300405f9c9357c40351b257" candidate_uuid: "aae31ac2fd8f437094da0fec2001e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed81f01909134a5e9017894fcf342a30"
I20260928 02:57:08.459945 14097 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708200b16300405f9c9357c40351b257" candidate_uuid: "aae31ac2fd8f437094da0fec2001e7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a40171df67487eb48c41737d5f5ec7"
I20260928 02:57:08.460060 13966 raft_consensus.cc:3037] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:08.460186 14097 raft_consensus.cc:3037] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:08.462563 14097 raft_consensus.cc:2454] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae31ac2fd8f437094da0fec2001e7ed in term 1.
I20260928 02:57:08.462604 13966 raft_consensus.cc:2454] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aae31ac2fd8f437094da0fec2001e7ed in term 1.
I20260928 02:57:08.463016 14160 leader_election.cc:302] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9a40171df67487eb48c41737d5f5ec7, aae31ac2fd8f437094da0fec2001e7ed; no voters:
I20260928 02:57:08.463289 14298 raft_consensus.cc:2781] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:08.463708 14298 raft_consensus.cc:686] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 1 LEADER]: Becoming Leader. State: Replica: aae31ac2fd8f437094da0fec2001e7ed, State: Running, Role: LEADER
I20260928 02:57:08.464169 14298 consensus_queue.cc:237] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } }
I20260928 02:57:08.468588 13823 catalog_manager.cc:5495] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed reported cstate change: term changed from 0 to 1, leader changed from <none> to aae31ac2fd8f437094da0fec2001e7ed (127.27.229.195). New cstate: current_term: 1 leader_uuid: "aae31ac2fd8f437094da0fec2001e7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aae31ac2fd8f437094da0fec2001e7ed" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 34469 } health_report { overall_health: HEALTHY } } }
I20260928 02:57:08.524663 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.518688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49568 (local address 127.27.229.195:34469)
0928 02:57:08.518914 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.518919 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.518938 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:57:08.519199 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.519203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.519285 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.519443 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:08.519827 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.520555 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.522175 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.522637 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.522739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.523466 (+ 727us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:08.524018 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:08.524133 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.524278 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.524345 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 02:57:08.533200 13966 raft_consensus.cc:1261] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 1 FOLLOWER]: Refusing update from remote peer aae31ac2fd8f437094da0fec2001e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:08.533200 14097 raft_consensus.cc:1261] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 1 FOLLOWER]: Refusing update from remote peer aae31ac2fd8f437094da0fec2001e7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:08.534474 14298 consensus_queue.cc:1035] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed81f01909134a5e9017894fcf342a30" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 40427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:08.534893 14300 consensus_queue.cc:1035] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a40171df67487eb48c41737d5f5ec7" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 40891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:08.566397 13824 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:08.566704 13824 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45976:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:08.570873 13824 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 483b206fd6524dabb52d3c5431e05181: Sending DeleteTablet for 3 replicas of tablet 708200b16300405f9c9357c40351b257
I20260928 02:57:08.572196 13946 tablet_service.cc:1514] Processing DeleteTablet for tablet 708200b16300405f9c9357c40351b257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:08 UTC) from {username='slave'} at 127.0.0.1:37754
I20260928 02:57:08.572175 14077 tablet_service.cc:1514] Processing DeleteTablet for tablet 708200b16300405f9c9357c40351b257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:08 UTC) from {username='slave'} at 127.0.0.1:35798
I20260928 02:57:08.572470 14207 tablet_service.cc:1514] Processing DeleteTablet for tablet 708200b16300405f9c9357c40351b257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:08 UTC) from {username='slave'} at 127.0.0.1:59398
I20260928 02:57:08.574497 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13884
I20260928 02:57:08.584690 14314 tablet_replica.cc:330] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: stopping tablet replica
I20260928 02:57:08.584928 14315 tablet_replica.cc:330] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: stopping tablet replica
I20260928 02:57:08.585541 14315 raft_consensus.cc:2229] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:08.585932 14314 raft_consensus.cc:2229] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:08.586480 14315 raft_consensus.cc:2258] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:08.587023 14314 raft_consensus.cc:2258] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:08.589331 14315 ts_tablet_manager.cc:1905] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:08.590199 14313 tablet_replica.cc:330] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: stopping tablet replica
I20260928 02:57:08.590602 14313 raft_consensus.cc:2229] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:08.591193 14313 raft_consensus.cc:2258] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:08.593380 14313 ts_tablet_manager.cc:1905] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:08.597000 14315 ts_tablet_manager.cc:1918] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:08.597334 14315 log.cc:1196] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/708200b16300405f9c9357c40351b257
I20260928 02:57:08.597977 14315 ts_tablet_manager.cc:1939] T 708200b16300405f9c9357c40351b257 P aae31ac2fd8f437094da0fec2001e7ed: Deleting consensus metadata
I20260928 02:57:08.599130 14314 ts_tablet_manager.cc:1905] T 708200b16300405f9c9357c40351b257 P ed81f01909134a5e9017894fcf342a30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:08.599252 14284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.575698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.27.229.193:40427)
0928 02:57:08.576038 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.576044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.579313 (+ 3269us) server_negotiation.cc:407] Connection header received
0928 02:57:08.579524 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.579529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.579646 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.579748 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:08.591529 (+ 11781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.592697 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.594916 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.595588 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.595689 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.597275 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:08.597288 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:08.597306 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:08.598166 (+ 860us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:08.598229 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:08.598255 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:08.598395 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.598756 (+ 361us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.598901 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260928 02:57:08.599920 13811 catalog_manager.cc:4841] TS aae31ac2fd8f437094da0fec2001e7ed (127.27.229.195:34469): tablet 708200b16300405f9c9357c40351b257 (table test-table [id=7f3cf3a079694c45853ef227206f5d30]) successfully deleted
I20260928 02:57:08.600659 14006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:08.600689 14313 ts_tablet_manager.cc:1918] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:08.600823 14313 log.cc:1196] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/708200b16300405f9c9357c40351b257
I20260928 02:57:08.601377 14313 ts_tablet_manager.cc:1939] T 708200b16300405f9c9357c40351b257 P a9a40171df67487eb48c41737d5f5ec7: Deleting consensus metadata
I20260928 02:57:08.603317 13809 catalog_manager.cc:4841] TS a9a40171df67487eb48c41737d5f5ec7 (127.27.229.194:40891): tablet 708200b16300405f9c9357c40351b257 (table test-table [id=7f3cf3a079694c45853ef227206f5d30]) successfully deleted
I20260928 02:57:08.696424 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13884
I20260928 02:57:08.715763 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14015
W20260928 02:57:08.715656 13810 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.229.193:40427 (error 108)
I20260928 02:57:08.717803 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.716915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52762)
0928 02:57:08.717417 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.717602 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":300,"threads_started":1}
W20260928 02:57:08.718117 13810 catalog_manager.cc:4584] TS ed81f01909134a5e9017894fcf342a30 (127.27.229.193:40427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 708200b16300405f9c9357c40351b257: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
I20260928 02:57:08.724627 14288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.717007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41044 (local address 127.27.229.194:40891)
0928 02:57:08.717232 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.717238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.717258 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:57:08.717326 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.717332 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.717449 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.717560 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:08.718110 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.719252 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.720688 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.721367 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.721499 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.722622 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:08.722634 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:08.722653 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:08.723700 (+ 1047us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:08.723762 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:08.723788 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:08.723905 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.724113 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.724246 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 02:57:08.725342 14137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:08.772060 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.771496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52778)
0928 02:57:08.771758 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.771877 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260928 02:57:08.773296 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.772855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52784)
0928 02:57:08.773037 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.773125 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:57:08.818800 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14015
I20260928 02:57:08.834004 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.833399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52794)
0928 02:57:08.833654 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.833773 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:57:08.835348 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.834891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52804)
0928 02:57:08.835068 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.835165 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 02:57:08.837548 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14145
I20260928 02:57:08.846079 14289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.838571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49584 (local address 127.27.229.195:34469)
0928 02:57:08.838779 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:08.838785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:08.838807 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:57:08.838902 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:08.838908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:08.839014 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:08.839130 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:08.839544 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.840749 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.842090 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:08.842803 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:08.843003 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:08.844311 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:08.844323 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:08.844344 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:08.845203 (+ 859us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:08.845267 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:08.845296 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:08.845414 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:08.845587 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:57:08.845698 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:57:08.846930 14267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:08.916143 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.915672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52806)
0928 02:57:08.915913 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.915998 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 02:57:08.917258 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:08.916834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52816)
0928 02:57:08.917065 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:08.917133 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 02:57:08.972434 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14145
I20260928 02:57:08.992467 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 13794
I20260928 02:57:08.993609 13855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:09.106346 14319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:09.105545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:40427 (local address 127.0.0.1:52820)
0928 02:57:09.105947 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:09.106096 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:40427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193}
I20260928 02:57:09.106618 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 13794
2026-09-28T02:57:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3122 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:09 dist-test-slave-s1kg krb5kdc[14325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:09 dist-test-slave-s1kg krb5kdc[14325](info): set up 2 sockets
Sep 28 02:57:09 dist-test-slave-s1kg krb5kdc[14325](info): commencing operation
krb5kdc: starting...
W20260928 02:57:10.238924 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:10 dist-test-slave-s1kg krb5kdc[14325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:10.497824 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41885
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:41885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:10.835580 14338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:10.835947 14338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:10.836107 14338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:10.844347 14338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:10.844494 14338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:10.844574 14338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:10.844655 14338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:10.844724 14338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:10.855296 14338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:41885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:41885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:10.856696 14338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:10.857993 14338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:10.870365 14343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:10.870431 14344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:10.870481 14346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:10.871455 14338 server_base.cc:1034] running on GCE node
I20260928 02:57:10.872217 14338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:10.873741 14338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:10.874950 14338 hybrid_clock.cc:648] HybridClock initialized: now 1790564230874893 us; error 32 us; skew 500 ppm
I20260928 02:57:10.875267 14338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:10 dist-test-slave-s1kg krb5kdc[14325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:10.881574 14338 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:10.883309 14338 webserver.cc:460] Webserver started at http://127.27.229.254:35787/ using document root <none> and password file <none>
I20260928 02:57:10.884055 14338 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:10.884189 14338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:10.884539 14338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:10.887650 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "d0bda288cc4d4777ac874424c8ef0a63"
format_stamp: "Formatted at 2026-09-28 02:57:10 on dist-test-slave-s1kg"
server_key: "d69203b698009fd4c0a0f3c40645e9d2"
server_key_iv: "95f3eed85190a5793a921cbfaf5ded6d"
server_key_version: "encryptionkey@0"
I20260928 02:57:10.888449 14338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "d0bda288cc4d4777ac874424c8ef0a63"
format_stamp: "Formatted at 2026-09-28 02:57:10 on dist-test-slave-s1kg"
server_key: "d69203b698009fd4c0a0f3c40645e9d2"
server_key_iv: "95f3eed85190a5793a921cbfaf5ded6d"
server_key_version: "encryptionkey@0"
I20260928 02:57:10.893731 14338 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 02:57:10.897231 14353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:10.898563 14338 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 02:57:10.898756 14338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "d0bda288cc4d4777ac874424c8ef0a63"
format_stamp: "Formatted at 2026-09-28 02:57:10 on dist-test-slave-s1kg"
server_key: "d69203b698009fd4c0a0f3c40645e9d2"
server_key_iv: "95f3eed85190a5793a921cbfaf5ded6d"
server_key_version: "encryptionkey@0"
I20260928 02:57:10.899034 14338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:10.912266 14338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:10.916754 14338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:10.917064 14338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:10.935981 14338 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:41885
I20260928 02:57:10.936004 14404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:41885 every 8 connection(s)
I20260928 02:57:10.937420 14338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:10.941684 14405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:10.942965 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14338
I20260928 02:57:10.943163 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:10.943615 28567 external_mini_cluster.cc:1428] Setting key fcb8299cb22ab5feea8ad9ee2c6fc3f8
I20260928 02:57:10.958070 14405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: Bootstrap starting.
I20260928 02:57:10.962047 14405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:10.963461 14405 log.cc:824] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:57:10 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:10.966629 14405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: No bootstrap required, opened a new log
I20260928 02:57:10.973479 14408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:10.945633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50808 (local address 127.27.229.254:41885)
0928 02:57:10.946990 (+ 1357us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:10.947007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:10.947067 (+ 60us) server_negotiation.cc:407] Connection header received
0928 02:57:10.949423 (+ 2356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:10.949452 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:10.950010 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:10.950478 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:10.951861 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:10.953900 (+ 2039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:10.955985 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:10.957074 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:10.957265 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:10.965649 (+ 8384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:10.965657 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:10.965672 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:10.965710 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:10.969105 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:10.969979 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:10.969986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:10.969995 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:10.970154 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:10.970571 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:10.970575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:10.970577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:10.971161 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:10.971408 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:10.971936 (+ 528us) server_negotiation.cc:299] Negotiation successful
0928 02:57:10.972370 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1077,"thread_start_us":837,"threads_started":1}
I20260928 02:57:10.975543 14405 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } }
I20260928 02:57:10.975929 14405 raft_consensus.cc:374] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:10.976054 14405 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0bda288cc4d4777ac874424c8ef0a63, State: Initialized, Role: FOLLOWER
I20260928 02:57:10.976677 14405 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } }
I20260928 02:57:10.976897 14405 raft_consensus.cc:388] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:10.976995 14405 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:10.977116 14405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:10.979334 14405 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } }
I20260928 02:57:10.979883 14405 leader_election.cc:302] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d0bda288cc4d4777ac874424c8ef0a63; no voters:
I20260928 02:57:10.980614 14405 leader_election.cc:288] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:10.980736 14410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:10.981468 14410 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [term 1 LEADER]: Becoming Leader. State: Replica: d0bda288cc4d4777ac874424c8ef0a63, State: Running, Role: LEADER
I20260928 02:57:10.982072 14410 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } }
I20260928 02:57:10.983083 14405 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:10.985270 14411 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0bda288cc4d4777ac874424c8ef0a63. Latest consensus state: current_term: 1 leader_uuid: "d0bda288cc4d4777ac874424c8ef0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } } }
I20260928 02:57:10.985340 14412 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d0bda288cc4d4777ac874424c8ef0a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d0bda288cc4d4777ac874424c8ef0a63" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 41885 } } }
I20260928 02:57:10.985632 14411 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:10.985716 14412 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:10.992164 14416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:10.997118 14416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:11.006103 14416 catalog_manager.cc:1347] Generated new cluster ID: e8d78370fa9948109de95dc5fbd04229
I20260928 02:57:11.006239 14416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:11.022974 14416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:11.024966 14416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:11.036317 14416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: Generated new TSK 0
I20260928 02:57:11.037261 14416 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:57:11.106477 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:11.457930 14431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:11.458197 14431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:11.458292 14431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:11.465932 14431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:11.466041 14431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:11.466157 14431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:11.476449 14431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:11.477707 14431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:11.479039 14431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:11.493151 14437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:11.493184 14439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:11.493119 14436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:11.493876 14431 server_base.cc:1034] running on GCE node
I20260928 02:57:11.494508 14431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:11.495569 14431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:11.496837 14431 hybrid_clock.cc:648] HybridClock initialized: now 1790564231496773 us; error 50 us; skew 500 ppm
I20260928 02:57:11.497197 14431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:11 dist-test-slave-s1kg krb5kdc[14325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564231, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:11.500540 14431 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:57:11.502146 14431 webserver.cc:460] Webserver started at http://127.27.229.193:41533/ using document root <none> and password file <none>
I20260928 02:57:11.502977 14431 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:11.503124 14431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:11.503445 14431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:11.506458 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "71dac0eef81144bca823cdca538fb928"
format_stamp: "Formatted at 2026-09-28 02:57:11 on dist-test-slave-s1kg"
server_key: "6cb77672c0935eba377f04c81322588a"
server_key_iv: "832bd163685c79295c8581ae67fd3055"
server_key_version: "encryptionkey@0"
I20260928 02:57:11.507289 14431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "71dac0eef81144bca823cdca538fb928"
format_stamp: "Formatted at 2026-09-28 02:57:11 on dist-test-slave-s1kg"
server_key: "6cb77672c0935eba377f04c81322588a"
server_key_iv: "832bd163685c79295c8581ae67fd3055"
server_key_version: "encryptionkey@0"
I20260928 02:57:11.512657 14431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:57:11.516641 14446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:11.518134 14431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:57:11.518334 14431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "71dac0eef81144bca823cdca538fb928"
format_stamp: "Formatted at 2026-09-28 02:57:11 on dist-test-slave-s1kg"
server_key: "6cb77672c0935eba377f04c81322588a"
server_key_iv: "832bd163685c79295c8581ae67fd3055"
server_key_version: "encryptionkey@0"
I20260928 02:57:11.518559 14431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:11.542212 14431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:11.546734 14431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:11.547066 14431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:11.548100 14431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:11.549486 14431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:11.549588 14431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:11.549739 14431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:11.549811 14431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:11.584270 14431 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:39359
I20260928 02:57:11.584357 14558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:39359 every 8 connection(s)
I20260928 02:57:11.585616 14431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:11.592105 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14431
I20260928 02:57:11.592303 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:11.592824 28567 external_mini_cluster.cc:1428] Setting key 469d5c58eab974901d552ee2390872a0
Sep 28 02:57:11 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564231, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:11.609184 14561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:11.588298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:53497 (local address 127.27.229.254:41885)
0928 02:57:11.588769 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:11.588773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:11.589841 (+ 1068us) server_negotiation.cc:407] Connection header received
0928 02:57:11.591814 (+ 1973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:11.591820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:11.591932 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:11.592064 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:11.593814 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:11.594977 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:11.597842 (+ 2865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:11.598260 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:11.598371 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:11.603093 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:11.603099 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:11.603102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:11.603139 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:11.605963 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:11.606752 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:11.606758 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:11.606762 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:11.606945 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:11.607394 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:11.607400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:11.607403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:11.607679 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:11.607843 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:11.608588 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 02:57:11.608822 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":246,"threads_started":1}
I20260928 02:57:11.610229 14562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:11.588837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41885 (local address 127.27.229.193:53497)
0928 02:57:11.589648 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:11.589696 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:11.591519 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:11.592345 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:11.592354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:11.592949 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:11.593582 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:11.593608 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:11.595142 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:11.595147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:11.597654 (+ 2507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:11.597675 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:11.598535 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:11.598544 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:11.598928 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:11.599761 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:11.599799 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:11.602792 (+ 2993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:11.606129 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:11.606139 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:11.606158 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:11.606619 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:11.607061 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:11.607066 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:11.607070 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:11.607267 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:11.607825 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:11.607833 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:11.608206 (+ 373us) client_negotiation.cc:769] Sending connection context
0928 02:57:11.608489 (+ 283us) client_negotiation.cc:240] Negotiation successful
0928 02:57:11.609062 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":307,"threads_started":1}
I20260928 02:57:11.611927 14559 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41885
I20260928 02:57:11.612290 14559 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:11.613178 14559 heartbeater.cc:511] Master 127.27.229.254:41885 requested a full tablet report, sending...
I20260928 02:57:11.615814 14370 ts_manager.cc:194] Registered new tserver with Master: 71dac0eef81144bca823cdca538fb928 (127.27.229.193:39359)
I20260928 02:57:11.618187 14370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:53497
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:57:11.654750 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:11.989224 14565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:11.989535 14565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:11.989634 14565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:11.997107 14565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:11.997232 14565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:11.997347 14565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:12.007773 14565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:12.009058 14565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:12.010371 14565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:12.024904 14571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:12.024997 14573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:12.025024 14570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:12.026973 14565 server_base.cc:1034] running on GCE node
I20260928 02:57:12.027619 14565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:12.028602 14565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:12.029815 14565 hybrid_clock.cc:648] HybridClock initialized: now 1790564232029755 us; error 60 us; skew 500 ppm
I20260928 02:57:12.030205 14565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:12 dist-test-slave-s1kg krb5kdc[14325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564232, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:12.033579 14565 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:57:12.035282 14565 webserver.cc:460] Webserver started at http://127.27.229.194:46151/ using document root <none> and password file <none>
I20260928 02:57:12.035995 14565 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:12.036094 14565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:12.036386 14565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:12.039777 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "f0871d3b066e4664bf9a30ab9e928d82"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "498e1df4678d22a20732b20c0406a582"
server_key_iv: "3a7f8dac92016c7754303f9945fab3db"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.040568 14565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "f0871d3b066e4664bf9a30ab9e928d82"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "498e1df4678d22a20732b20c0406a582"
server_key_iv: "3a7f8dac92016c7754303f9945fab3db"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.045977 14565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:57:12.049839 14580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.051667 14565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:12.051880 14565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "f0871d3b066e4664bf9a30ab9e928d82"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "498e1df4678d22a20732b20c0406a582"
server_key_iv: "3a7f8dac92016c7754303f9945fab3db"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.052109 14565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:12.074151 14565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:12.078810 14565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:12.079176 14565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:12.080319 14565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:12.081763 14565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:12.081872 14565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.082023 14565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:12.082100 14565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.118388 14565 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:34123
I20260928 02:57:12.118498 14692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:34123 every 8 connection(s)
I20260928 02:57:12.119913 14565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:12.120841 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14565
I20260928 02:57:12.121043 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:12.121614 28567 external_mini_cluster.cc:1428] Setting key 63a437de4da708882d1898262e2c8fa8
Sep 28 02:57:12 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564232, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:12.146548 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.123086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:48625 (local address 127.27.229.254:41885)
0928 02:57:12.123547 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.123560 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.124990 (+ 1430us) server_negotiation.cc:407] Connection header received
0928 02:57:12.128092 (+ 3102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.128097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.128184 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.128287 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:12.130661 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.131556 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.134232 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.134683 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.134836 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.138984 (+ 4148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:12.138993 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:12.139001 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:12.139051 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:12.142889 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.143763 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.143771 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.143777 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.143941 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.144404 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.144412 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.144418 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.144826 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:12.145000 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.145745 (+ 745us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.146038 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":255,"threads_started":1}
I20260928 02:57:12.147228 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.123826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41885 (local address 127.27.229.194:48625)
0928 02:57:12.124759 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.124819 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.127704 (+ 2885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.128596 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.128612 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.129481 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:12.130372 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.130408 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.131742 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.131751 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.134049 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.134072 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.134981 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.134989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.135257 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.136027 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:12.136051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:12.138657 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:12.143070 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:12.143078 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:12.143093 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:12.143614 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:12.144069 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:12.144072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:12.144074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:12.144285 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:12.144967 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:12.144973 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:12.145338 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.145597 (+ 259us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.146026 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":339,"threads_started":1}
I20260928 02:57:12.148916 14693 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41885
I20260928 02:57:12.149302 14693 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:12.150259 14693 heartbeater.cc:511] Master 127.27.229.254:41885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
I20260928 02:57:12.152238 14370 ts_manager.cc:194] Registered new tserver with Master: f0871d3b066e4664bf9a30ab9e928d82 (127.27.229.194:34123)
I20260928 02:57:12.153146 14370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:48625
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:57:12.184387 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:35413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:12.518479 14699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:12.518822 14699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:12.519023 14699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:12.526801 14699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:12.526983 14699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:12.527205 14699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:12.537487 14699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:35413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:41885
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:12.538910 14699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:12.540237 14699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:12.554332 14705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:12.554376 14704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:12.554432 14707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:12.555243 14699 server_base.cc:1034] running on GCE node
I20260928 02:57:12.555994 14699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:12.557123 14699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:12.558403 14699 hybrid_clock.cc:648] HybridClock initialized: now 1790564232558342 us; error 55 us; skew 500 ppm
I20260928 02:57:12.558959 14699 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:12 dist-test-slave-s1kg krb5kdc[14325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564232, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:12.562371 14699 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:57:12.564162 14699 webserver.cc:460] Webserver started at http://127.27.229.195:41799/ using document root <none> and password file <none>
I20260928 02:57:12.564922 14699 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:12.565078 14699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:12.565397 14699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:12.568544 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "714797fd041b4fe99c3a92f54cae8f03"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "5238eaac5f8d02be533833ef04dabb9b"
server_key_iv: "28a6307bbf1f4ce7bf90b9167ffc9f33"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.569362 14699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "714797fd041b4fe99c3a92f54cae8f03"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "5238eaac5f8d02be533833ef04dabb9b"
server_key_iv: "28a6307bbf1f4ce7bf90b9167ffc9f33"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.574962 14699 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260928 02:57:12.578547 14714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.580029 14699 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 02:57:12.580242 14699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "714797fd041b4fe99c3a92f54cae8f03"
format_stamp: "Formatted at 2026-09-28 02:57:12 on dist-test-slave-s1kg"
server_key: "5238eaac5f8d02be533833ef04dabb9b"
server_key_iv: "28a6307bbf1f4ce7bf90b9167ffc9f33"
server_key_version: "encryptionkey@0"
I20260928 02:57:12.580520 14699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:12.607254 14699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:12.612103 14699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:12.612439 14699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:12.613526 14699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:12.615053 14699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:12.615161 14699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.615314 14699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:12.615393 14699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:12.622258 14559 heartbeater.cc:503] Master 127.27.229.254:41885 was elected leader, sending a full tablet report...
I20260928 02:57:12.652235 14699 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:39999
I20260928 02:57:12.652292 14826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:39999 every 8 connection(s)
I20260928 02:57:12.653654 14699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:12.659713 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14699
I20260928 02:57:12.659906 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:12.660507 28567 external_mini_cluster.cc:1428] Setting key 7812c08675a72894791219c52ef091b1
Sep 28 02:57:12 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564232, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:12.675299 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.656243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:60019 (local address 127.27.229.254:41885)
0928 02:57:12.656794 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.656798 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.657794 (+ 996us) server_negotiation.cc:407] Connection header received
0928 02:57:12.659850 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.659856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.659979 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.660112 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:12.661907 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.663018 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.664898 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.665315 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.665421 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.668956 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:12.668961 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:12.668964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:12.668998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:12.671752 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.672557 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.672563 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.672567 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.672699 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.673182 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.673188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.673191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.673522 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:12.673695 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.674597 (+ 902us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.674879 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":296,"threads_started":1}
I20260928 02:57:12.676334 14830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.656769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:41885 (local address 127.27.229.195:60019)
0928 02:57:12.657603 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.657649 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.659514 (+ 1865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.660330 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.660339 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.660966 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:12.661746 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.661767 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.663232 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.663240 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.664722 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.664749 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.665538 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.665544 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.665747 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.666416 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:12.666438 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:12.668723 (+ 2285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:12.671917 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:12.671925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:12.671941 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:12.672402 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:12.672846 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:12.672850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:12.672854 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:12.673061 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:12.673661 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:12.673670 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:12.674098 (+ 428us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.674475 (+ 377us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.675137 (+ 662us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":347,"threads_started":1}
I20260928 02:57:12.677750 14827 heartbeater.cc:347] Connected to a master server at 127.27.229.254:41885
I20260928 02:57:12.678077 14827 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:12.678812 14827 heartbeater.cc:511] Master 127.27.229.254:41885 requested a full tablet report, sending...
I20260928 02:57:12.680280 14370 ts_manager.cc:194] Registered new tserver with Master: 714797fd041b4fe99c3a92f54cae8f03 (127.27.229.195:39999)
I20260928 02:57:12.681061 14370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:60019
I20260928 02:57:12.688711 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:12.707090 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.693087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50812 (local address 127.27.229.254:41885)
0928 02:57:12.693347 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.693357 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.693727 (+ 370us) server_negotiation.cc:407] Connection header received
0928 02:57:12.693881 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.693885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.693965 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.694053 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:12.695673 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.696422 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.697833 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.698241 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.698364 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.701222 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:12.701227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:12.701231 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:12.701267 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:12.704112 (+ 2845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.705000 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.705004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.705007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.705129 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:12.705630 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:12.705633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:12.705635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:12.705949 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:12.706158 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.706521 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.706669 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:57:12.711269 14370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:12.718252 14370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:12.749800 14840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.731231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:34123 (local address 127.0.0.1:51580)
0928 02:57:12.738615 (+ 7384us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.738677 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.738986 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.740325 (+ 1339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.740336 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.740382 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:12.740949 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.740967 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.743287 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.743295 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.745705 (+ 2410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.745720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.748312 (+ 2592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.748318 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.749091 (+ 773us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.749194 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.749321 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.749480 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7014,"mutex_wait_us":496,"thread_start_us":11657,"threads_started":1}
I20260928 02:57:12.750576 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.731292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.27.229.194:34123)
0928 02:57:12.737914 (+ 6622us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.737926 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.739253 (+ 1327us) server_negotiation.cc:407] Connection header received
0928 02:57:12.739400 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.739412 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.739752 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.739980 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:12.741203 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.743104 (+ 1901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.745895 (+ 2791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.747795 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.747978 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.748165 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.750104 (+ 1939us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.750302 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6448,"thread_start_us":6339,"threads_started":1}
I20260928 02:57:12.751725 14841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.731257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:39999 (local address 127.0.0.1:46318)
0928 02:57:12.743222 (+ 11965us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.743254 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.743392 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.744172 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.744177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.744203 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:12.744633 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.744649 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.746603 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.746608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.749077 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.749093 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.750979 (+ 1886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.750984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.751102 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.751125 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.751199 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.751289 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11815,"mutex_wait_us":121,"thread_start_us":6816,"threads_started":1}
I20260928 02:57:12.751756 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.731639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46318 (local address 127.27.229.195:39999)
0928 02:57:12.742953 (+ 11314us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.742970 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.743549 (+ 579us) server_negotiation.cc:407] Connection header received
0928 02:57:12.743662 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.743670 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.743928 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.744736 (+ 808us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:12.745031 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.746442 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.749238 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.750739 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.750967 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.751140 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.751245 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.751412 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11082,"thread_start_us":10899,"threads_started":1}
I20260928 02:57:12.753600 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.741985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:39359 (local address 127.0.0.1:43590)
0928 02:57:12.743594 (+ 1609us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.743613 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.743819 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.744745 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.744748 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.744771 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:12.745243 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.745257 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.746769 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.746777 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.749162 (+ 2385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.749178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.752783 (+ 3605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.752790 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.752969 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.753010 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.753108 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.753234 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1484,"mutex_wait_us":63,"thread_start_us":1298,"threads_started":1}
I20260928 02:57:12.754654 14628 tablet_service.cc:1467] Processing CreateTablet for tablet 8137718bad8943e2ac3228f2fab95c5b (DEFAULT_TABLE table=test-table [id=c5fdccee76994a09b879e5f4c2491dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:12.755183 14762 tablet_service.cc:1467] Processing CreateTablet for tablet 8137718bad8943e2ac3228f2fab95c5b (DEFAULT_TABLE table=test-table [id=c5fdccee76994a09b879e5f4c2491dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:12.756283 14628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8137718bad8943e2ac3228f2fab95c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:12.756516 14762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8137718bad8943e2ac3228f2fab95c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:12.759614 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.739721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43590 (local address 127.27.229.193:39359)
0928 02:57:12.741923 (+ 2202us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.741932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.743717 (+ 1785us) server_negotiation.cc:407] Connection header received
0928 02:57:12.744017 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.744024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.744260 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.744466 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:12.745435 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.746600 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.749307 (+ 2707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.752575 (+ 3268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.752824 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.753015 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.759011 (+ 5996us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.759217 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2042,"thread_start_us":1871,"threads_started":1}
I20260928 02:57:12.763540 14494 tablet_service.cc:1467] Processing CreateTablet for tablet 8137718bad8943e2ac3228f2fab95c5b (DEFAULT_TABLE table=test-table [id=c5fdccee76994a09b879e5f4c2491dfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:12.765146 14494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8137718bad8943e2ac3228f2fab95c5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:12.769374 14846 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Bootstrap starting.
I20260928 02:57:12.771610 14847 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Bootstrap starting.
I20260928 02:57:12.773399 14846 tablet_bootstrap.cc:654] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:12.774747 14846 log.cc:824] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:12.775228 14847 tablet_bootstrap.cc:654] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:12.776793 14847 log.cc:824] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:12.779160 14846 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: No bootstrap required, opened a new log
I20260928 02:57:12.779554 14846 ts_tablet_manager.cc:1397] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20260928 02:57:12.779850 14850 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Bootstrap starting.
I20260928 02:57:12.780037 14847 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: No bootstrap required, opened a new log
I20260928 02:57:12.780313 14847 ts_tablet_manager.cc:1397] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 02:57:12.783062 14850 tablet_bootstrap.cc:654] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:12.784441 14850 log.cc:824] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:12.788389 14850 tablet_bootstrap.cc:492] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: No bootstrap required, opened a new log
I20260928 02:57:12.788720 14850 ts_tablet_manager.cc:1397] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.007s
I20260928 02:57:12.791589 14847 raft_consensus.cc:348] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.791589 14846 raft_consensus.cc:348] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.792092 14847 raft_consensus.cc:374] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:12.792088 14846 raft_consensus.cc:374] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:12.792196 14846 raft_consensus.cc:729] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 714797fd041b4fe99c3a92f54cae8f03, State: Initialized, Role: FOLLOWER
I20260928 02:57:12.792192 14847 raft_consensus.cc:729] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0871d3b066e4664bf9a30ab9e928d82, State: Initialized, Role: FOLLOWER
I20260928 02:57:12.792882 14846 consensus_queue.cc:260] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.792876 14847 consensus_queue.cc:260] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.794499 14693 heartbeater.cc:503] Master 127.27.229.254:41885 was elected leader, sending a full tablet report...
I20260928 02:57:12.794920 14846 ts_tablet_manager.cc:1428] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 02:57:12.795073 14847 ts_tablet_manager.cc:1428] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260928 02:57:12.795440 14827 heartbeater.cc:503] Master 127.27.229.254:41885 was elected leader, sending a full tablet report...
I20260928 02:57:12.798220 14850 raft_consensus.cc:348] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.798744 14850 raft_consensus.cc:374] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:12.798911 14850 raft_consensus.cc:729] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71dac0eef81144bca823cdca538fb928, State: Initialized, Role: FOLLOWER
I20260928 02:57:12.799521 14850 consensus_queue.cc:260] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.801376 14850 ts_tablet_manager.cc:1428] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Time spent starting tablet: real 0.013s user 0.012s sys 0.002s
W20260928 02:57:12.838202 14560 tablet.cc:2369] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:12.841073 14854 raft_consensus.cc:482] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:12.841375 14854 raft_consensus.cc:504] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.843225 14854 leader_election.cc:288] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0871d3b066e4664bf9a30ab9e928d82 (127.27.229.194:34123), 714797fd041b4fe99c3a92f54cae8f03 (127.27.229.195:39999)
I20260928 02:57:12.850816 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.843806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:37801 (local address 127.27.229.194:34123)
0928 02:57:12.844050 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.844057 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.844732 (+ 675us) server_negotiation.cc:407] Connection header received
0928 02:57:12.844946 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.844951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.845026 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.845115 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:12.846069 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.847100 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.849011 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.849769 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.849953 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.850037 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.850370 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.850521 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:57:12.850800 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.843808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:36103 (local address 127.27.229.195:39999)
0928 02:57:12.844026 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.844035 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.844738 (+ 703us) server_negotiation.cc:407] Connection header received
0928 02:57:12.844886 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.844892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.844975 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.845090 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:12.846110 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.847089 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.848999 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.849786 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.849971 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.850056 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.850384 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.850519 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:57:12.851055 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.843405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:39999 (local address 127.27.229.193:36103)
0928 02:57:12.844550 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.844570 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.844741 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.845249 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.845254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.845284 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:12.845953 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.845967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.847238 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.847246 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.848834 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.848847 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.850055 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.850062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.850215 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.850239 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.850344 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.850520 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":34,"thread_start_us":900,"threads_started":1}
I20260928 02:57:12.851076 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.843586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:34123 (local address 127.27.229.193:37801)
0928 02:57:12.844609 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:12.844625 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:12.844810 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:12.845281 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:12.845285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:12.845301 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:12.845845 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.845867 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.847303 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.847307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.848838 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:12.848850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.850016 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.850022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:12.850200 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.850226 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:57:12.850341 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 02:57:12.850544 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"mutex_wait_us":164,"spinlock_wait_cycles":34048,"thread_start_us":431,"threads_started":1}
I20260928 02:57:12.852567 14782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8137718bad8943e2ac3228f2fab95c5b" candidate_uuid: "71dac0eef81144bca823cdca538fb928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714797fd041b4fe99c3a92f54cae8f03" is_pre_election: true
I20260928 02:57:12.852578 14648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8137718bad8943e2ac3228f2fab95c5b" candidate_uuid: "71dac0eef81144bca823cdca538fb928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0871d3b066e4664bf9a30ab9e928d82" is_pre_election: true
I20260928 02:57:12.853178 14782 raft_consensus.cc:2454] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71dac0eef81144bca823cdca538fb928 in term 0.
I20260928 02:57:12.853180 14648 raft_consensus.cc:2454] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71dac0eef81144bca823cdca538fb928 in term 0.
I20260928 02:57:12.853998 14449 leader_election.cc:302] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 714797fd041b4fe99c3a92f54cae8f03, 71dac0eef81144bca823cdca538fb928, f0871d3b066e4664bf9a30ab9e928d82; no voters:
I20260928 02:57:12.854494 14854 raft_consensus.cc:2781] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:12.854668 14854 raft_consensus.cc:482] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:12.854784 14854 raft_consensus.cc:3037] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:12.857069 14854 raft_consensus.cc:504] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.857683 14854 leader_election.cc:288] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [CANDIDATE]: Term 1 election: Requested vote from peers f0871d3b066e4664bf9a30ab9e928d82 (127.27.229.194:34123), 714797fd041b4fe99c3a92f54cae8f03 (127.27.229.195:39999)
I20260928 02:57:12.858230 14782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8137718bad8943e2ac3228f2fab95c5b" candidate_uuid: "71dac0eef81144bca823cdca538fb928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "714797fd041b4fe99c3a92f54cae8f03"
I20260928 02:57:12.858419 14782 raft_consensus.cc:3037] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:12.860695 14782 raft_consensus.cc:2454] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71dac0eef81144bca823cdca538fb928 in term 1.
I20260928 02:57:12.861078 14648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8137718bad8943e2ac3228f2fab95c5b" candidate_uuid: "71dac0eef81144bca823cdca538fb928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0871d3b066e4664bf9a30ab9e928d82"
I20260928 02:57:12.861143 14449 leader_election.cc:302] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 714797fd041b4fe99c3a92f54cae8f03, 71dac0eef81144bca823cdca538fb928; no voters:
I20260928 02:57:12.861299 14648 raft_consensus.cc:3037] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:12.861493 14854 raft_consensus.cc:2781] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:12.862010 14854 raft_consensus.cc:686] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 1 LEADER]: Becoming Leader. State: Replica: 71dac0eef81144bca823cdca538fb928, State: Running, Role: LEADER
I20260928 02:57:12.862603 14854 consensus_queue.cc:237] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } }
I20260928 02:57:12.864939 14648 raft_consensus.cc:2454] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71dac0eef81144bca823cdca538fb928 in term 1.
I20260928 02:57:12.868376 14369 catalog_manager.cc:5495] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 reported cstate change: term changed from 0 to 1, leader changed from <none> to 71dac0eef81144bca823cdca538fb928 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "71dac0eef81144bca823cdca538fb928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71dac0eef81144bca823cdca538fb928" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 39359 } health_report { overall_health: HEALTHY } } }
W20260928 02:57:12.872457 14694 tablet.cc:2369] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:12.905275 14828 tablet.cc:2369] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:12.915073 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.909474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43606 (local address 127.27.229.193:39359)
0928 02:57:12.909678 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.909683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.909699 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:57:12.909812 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.909815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.909904 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.910036 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:12.910481 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.911300 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.912769 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.913210 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.913350 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:12.913931 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:12.914437 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:12.914570 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:12.914757 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:57:12.914799 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:57:12.923590 14648 raft_consensus.cc:1261] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 1 FOLLOWER]: Refusing update from remote peer 71dac0eef81144bca823cdca538fb928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:12.923596 14782 raft_consensus.cc:1261] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 1 FOLLOWER]: Refusing update from remote peer 71dac0eef81144bca823cdca538fb928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:12.924592 14854 consensus_queue.cc:1035] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0871d3b066e4664bf9a30ab9e928d82" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 34123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:12.924921 14857 consensus_queue.cc:1035] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "714797fd041b4fe99c3a92f54cae8f03" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 39999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:12.938378 14860 mvcc.cc:205] Tried to move back new op lower bound from 7334151098043310080 to 7334151097810141184. Current Snapshot: MvccSnapshot[applied={T|T < 7334151098043310080}]
I20260928 02:57:12.977377 14369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:12.977736 14369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50812:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:12.982072 14369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d0bda288cc4d4777ac874424c8ef0a63: Sending DeleteTablet for 3 replicas of tablet 8137718bad8943e2ac3228f2fab95c5b
I20260928 02:57:12.983529 14762 tablet_service.cc:1514] Processing DeleteTablet for tablet 8137718bad8943e2ac3228f2fab95c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:12 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:46318
I20260928 02:57:12.983628 14628 tablet_service.cc:1514] Processing DeleteTablet for tablet 8137718bad8943e2ac3228f2fab95c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:12 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:51580
I20260928 02:57:12.983649 14494 tablet_service.cc:1514] Processing DeleteTablet for tablet 8137718bad8943e2ac3228f2fab95c5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:12 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:43590
I20260928 02:57:12.985364 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14431
I20260928 02:57:12.985790 14870 tablet_replica.cc:330] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: stopping tablet replica
I20260928 02:57:12.986452 14871 tablet_replica.cc:330] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: stopping tablet replica
I20260928 02:57:12.986617 14869 tablet_replica.cc:330] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: stopping tablet replica
I20260928 02:57:12.986975 14871 raft_consensus.cc:2229] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:12.987007 14869 raft_consensus.cc:2229] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:12.987538 14869 raft_consensus.cc:2258] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:12.987033 14870 raft_consensus.cc:2229] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:12.987732 14871 raft_consensus.cc:2258] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:12.988190 14870 raft_consensus.cc:2258] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:12.989913 14869 ts_tablet_manager.cc:1905] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:12.991588 14871 ts_tablet_manager.cc:1905] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:12.992195 14870 ts_tablet_manager.cc:1905] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:12.997707 14869 ts_tablet_manager.cc:1918] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:12.997910 14869 log.cc:1196] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/8137718bad8943e2ac3228f2fab95c5b
I20260928 02:57:12.998430 14869 ts_tablet_manager.cc:1939] T 8137718bad8943e2ac3228f2fab95c5b P 714797fd041b4fe99c3a92f54cae8f03: Deleting consensus metadata
I20260928 02:57:12.999943 14871 ts_tablet_manager.cc:1918] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:13.000139 14871 log.cc:1196] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/8137718bad8943e2ac3228f2fab95c5b
I20260928 02:57:13.000471 14356 catalog_manager.cc:4841] TS 714797fd041b4fe99c3a92f54cae8f03 (127.27.229.195:39999): tablet 8137718bad8943e2ac3228f2fab95c5b (table test-table [id=c5fdccee76994a09b879e5f4c2491dfa]) successfully deleted
I20260928 02:57:13.001535 14871 ts_tablet_manager.cc:1939] T 8137718bad8943e2ac3228f2fab95c5b P 71dac0eef81144bca823cdca538fb928: Deleting consensus metadata
I20260928 02:57:13.003461 14870 ts_tablet_manager.cc:1918] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:13.003639 14870 log.cc:1196] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/8137718bad8943e2ac3228f2fab95c5b
Sep 28 02:57:12 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:57:13.003693 14356 catalog_manager.cc:4841] TS 71dac0eef81144bca823cdca538fb928 (127.27.229.193:39359): tablet 8137718bad8943e2ac3228f2fab95c5b (table test-table [id=c5fdccee76994a09b879e5f4c2491dfa]) successfully deleted
I20260928 02:57:13.004307 14870 ts_tablet_manager.cc:1939] T 8137718bad8943e2ac3228f2fab95c5b P f0871d3b066e4664bf9a30ab9e928d82: Deleting consensus metadata
I20260928 02:57:13.006049 14357 catalog_manager.cc:4841] TS f0871d3b066e4664bf9a30ab9e928d82 (127.27.229.194:34123): tablet 8137718bad8943e2ac3228f2fab95c5b (table test-table [id=c5fdccee76994a09b879e5f4c2491dfa]) successfully deleted
I20260928 02:57:13.011718 14844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:12.988796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43618 (local address 127.27.229.193:39359)
0928 02:57:12.989036 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:12.989042 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:12.989063 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:57:12.989126 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:12.989131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:12.989213 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:12.989329 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:12.991115 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.992217 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.994194 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:12.995022 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:12.995135 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:13.005135 (+ 10000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:13.005152 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:13.005178 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:13.005229 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:13.008690 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.009471 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.009479 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.009482 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.009630 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.010035 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.010040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.010045 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.010431 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:13.010751 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:13.011101 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 02:57:13.011350 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:57:13.012495 14554 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:13.113291 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14431
I20260928 02:57:13.133615 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14565
Sep 28 02:57:13 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:57:13.149623 14842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:13.134469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51594 (local address 127.27.229.194:34123)
0928 02:57:13.134661 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:13.134667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:13.134715 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:57:13.134789 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:13.134794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:13.134877 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:13.134973 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:13.136192 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:13.136960 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:13.138283 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:13.138666 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:13.138774 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:13.143392 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:13.143405 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:13.143422 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:13.143459 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:13.146439 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.147479 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.147487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.147489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.147653 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.148037 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.148040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.148042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.148440 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:13.148665 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:13.148995 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:57:13.149183 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:57:13.150341 14688 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:13.240461 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14565
I20260928 02:57:13.257887 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14699
Sep 28 02:57:13 dist-test-slave-s1kg krb5kdc[14325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:57:13.273509 14843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:13.258659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46324 (local address 127.27.229.195:39999)
0928 02:57:13.258873 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:13.258882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:13.258904 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:57:13.259025 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:13.259029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:13.259097 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:13.259189 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:13.260544 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:13.261282 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:13.262592 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:13.263165 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:13.263264 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:13.267821 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:13.267837 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:13.267853 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:13.267894 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:13.270679 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.271496 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.271504 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.271506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.271654 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:13.272062 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:13.272066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:13.272068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:13.272471 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:13.272715 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:13.273062 (+ 347us) server_negotiation.cc:299] Negotiation successful
0928 02:57:13.273243 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:57:13.274168 14822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:13.366241 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14699
I20260928 02:57:13.383885 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14338
I20260928 02:57:13.385210 14400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:13.499876 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14338
2026-09-28T02:57:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4394 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T02:57:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:13Z Disabled control of system clock
I20260928 02:57:13.577834 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33877
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:33877
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:13.919899 14880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:13.920209 14880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:13.920293 14880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:13.927951 14880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:13.928076 14880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:13.928113 14880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:13.928164 14880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:13.928200 14880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:13.937763 14880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:33877
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:33877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:13.939056 14880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:13.940824 14880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:13.953589 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:13.953831 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:13.954178 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:13.954579 14880 server_base.cc:1034] running on GCE node
I20260928 02:57:13.955477 14880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:13.957188 14880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:13.958441 14880 hybrid_clock.cc:648] HybridClock initialized: now 1790564233958392 us; error 37 us; skew 500 ppm
I20260928 02:57:13.958968 14880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:13.961607 14880 webserver.cc:460] Webserver started at http://127.27.229.254:41849/ using document root <none> and password file <none>
I20260928 02:57:13.962383 14880 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:13.962479 14880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:13.962894 14880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:13.965991 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac"
format_stamp: "Formatted at 2026-09-28 02:57:13 on dist-test-slave-s1kg"
server_key: "527816f7361bfed8fbe7a3b5b5aff46e"
server_key_iv: "b8dcc0ba7625a1906fc1532bd8f7aeff"
server_key_version: "encryptionkey@0"
I20260928 02:57:13.966784 14880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac"
format_stamp: "Formatted at 2026-09-28 02:57:13 on dist-test-slave-s1kg"
server_key: "527816f7361bfed8fbe7a3b5b5aff46e"
server_key_iv: "b8dcc0ba7625a1906fc1532bd8f7aeff"
server_key_version: "encryptionkey@0"
I20260928 02:57:13.972280 14880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:57:13.976035 14894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:13.977532 14880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:57:13.977746 14880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac"
format_stamp: "Formatted at 2026-09-28 02:57:13 on dist-test-slave-s1kg"
server_key: "527816f7361bfed8fbe7a3b5b5aff46e"
server_key_iv: "b8dcc0ba7625a1906fc1532bd8f7aeff"
server_key_version: "encryptionkey@0"
I20260928 02:57:13.978015 14880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:14.003978 14880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:14.004928 14880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:14.005250 14880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:14.024827 14880 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:33877
I20260928 02:57:14.024835 14945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:33877 every 8 connection(s)
I20260928 02:57:14.026216 14880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:14.030037 14946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:14.034303 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14880
I20260928 02:57:14.034503 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:14.035020 28567 external_mini_cluster.cc:1428] Setting key 78523cdd1c31d4f2d1cd899f9f85de44
I20260928 02:57:14.041579 14946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: Bootstrap starting.
I20260928 02:57:14.044907 14946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:14.046002 14946 log.cc:824] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:14.049168 14946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: No bootstrap required, opened a new log
I20260928 02:57:14.049590 14949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:14.036926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44886 (local address 127.27.229.254:33877)
0928 02:57:14.037964 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:14.037975 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:14.038018 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:57:14.040345 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:14.040380 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:14.040873 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:14.041303 (+ 430us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:14.041924 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.043308 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.044772 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.045571 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.045778 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:14.047086 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:14.047097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:14.047124 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:14.047760 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:14.047836 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:14.047857 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:14.047953 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:14.048193 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 02:57:14.048574 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":370,"threads_started":1}
I20260928 02:57:14.057741 14946 raft_consensus.cc:348] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } }
I20260928 02:57:14.058161 14946 raft_consensus.cc:374] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:14.058270 14946 raft_consensus.cc:729] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daa7f6dcd54c4b89b0ae075d20f0b1ac, State: Initialized, Role: FOLLOWER
I20260928 02:57:14.058911 14946 consensus_queue.cc:260] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } }
I20260928 02:57:14.059163 14946 raft_consensus.cc:388] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:14.059306 14946 raft_consensus.cc:482] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:14.059491 14946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:14.061726 14946 raft_consensus.cc:504] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } }
I20260928 02:57:14.062259 14946 leader_election.cc:302] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: daa7f6dcd54c4b89b0ae075d20f0b1ac; no voters:
I20260928 02:57:14.063077 14946 leader_election.cc:288] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:14.063190 14951 raft_consensus.cc:2781] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:14.064253 14951 raft_consensus.cc:686] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [term 1 LEADER]: Becoming Leader. State: Replica: daa7f6dcd54c4b89b0ae075d20f0b1ac, State: Running, Role: LEADER
I20260928 02:57:14.064831 14951 consensus_queue.cc:237] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } }
I20260928 02:57:14.065606 14946 sys_catalog.cc:564] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:14.068527 14952 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } } }
I20260928 02:57:14.069036 14952 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:14.069586 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader daa7f6dcd54c4b89b0ae075d20f0b1ac. Latest consensus state: current_term: 1 leader_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "daa7f6dcd54c4b89b0ae075d20f0b1ac" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 33877 } } }
I20260928 02:57:14.069953 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:14.071344 14957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:14.078214 14957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:14.087543 14957 catalog_manager.cc:1347] Generated new cluster ID: 64098b2993b34f76bb01c86da14a7088
I20260928 02:57:14.087670 14957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:14.114758 14957 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:14.116781 14957 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:14.130084 14957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: Generated new TSK 0
I20260928 02:57:14.131260 14957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:14.139221 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--builtin_ntp_servers=127.27.229.212:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:14.498145 14970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:14.498498 14970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:14.498659 14970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:14.506456 14970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:14.506604 14970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:14.506783 14970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:14.516940 14970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:14.518270 14970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:14.520154 14970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:14.534029 14976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:14.534003 14978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:14.534088 14975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:14.534888 14970 server_base.cc:1034] running on GCE node
I20260928 02:57:14.535602 14970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:14.536711 14970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:14.537945 14970 hybrid_clock.cc:648] HybridClock initialized: now 1790564234537885 us; error 54 us; skew 500 ppm
I20260928 02:57:14.538276 14970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:14.540669 14970 webserver.cc:460] Webserver started at http://127.27.229.193:40103/ using document root <none> and password file <none>
I20260928 02:57:14.541379 14970 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:14.541517 14970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:14.541885 14970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:14.544929 14970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "3e31928d96be472d810c438ea69d4ab2"
format_stamp: "Formatted at 2026-09-28 02:57:14 on dist-test-slave-s1kg"
server_key: "2cc1e561d16382671f0b3dbbf13523fc"
server_key_iv: "80a227d456ac9f2d4abfce1d2945c9b1"
server_key_version: "encryptionkey@0"
I20260928 02:57:14.545766 14970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "3e31928d96be472d810c438ea69d4ab2"
format_stamp: "Formatted at 2026-09-28 02:57:14 on dist-test-slave-s1kg"
server_key: "2cc1e561d16382671f0b3dbbf13523fc"
server_key_iv: "80a227d456ac9f2d4abfce1d2945c9b1"
server_key_version: "encryptionkey@0"
I20260928 02:57:14.551457 14970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260928 02:57:14.555500 14984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:14.557044 14970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:14.557224 14970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "3e31928d96be472d810c438ea69d4ab2"
format_stamp: "Formatted at 2026-09-28 02:57:14 on dist-test-slave-s1kg"
server_key: "2cc1e561d16382671f0b3dbbf13523fc"
server_key_iv: "80a227d456ac9f2d4abfce1d2945c9b1"
server_key_version: "encryptionkey@0"
I20260928 02:57:14.557497 14970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:14.575672 14970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:14.576620 14970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:14.576942 14970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:14.578079 14970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:14.579756 14970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:14.579882 14970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:14.580013 14970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:14.580096 14970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:14.615320 14970 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:46733
I20260928 02:57:14.615361 15096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:46733 every 8 connection(s)
I20260928 02:57:14.616699 14970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:14.618371 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 14970
I20260928 02:57:14.618563 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:14.619120 28567 external_mini_cluster.cc:1428] Setting key 06ebcf4bfb49a84d35211791db1f09d6
I20260928 02:57:14.622265 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--builtin_ntp_servers=127.27.229.212:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:14.631247 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:14.619713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:52959 (local address 127.27.229.254:33877)
0928 02:57:14.620147 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:14.620152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:14.621675 (+ 1523us) server_negotiation.cc:407] Connection header received
0928 02:57:14.623778 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:14.623784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:14.623890 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:14.624017 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:14.625168 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.625897 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.627653 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.628082 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.628224 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:14.629954 (+ 1730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:14.629960 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:14.629964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:14.629986 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:14.630194 (+ 208us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:14.630223 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:14.630357 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:14.630710 (+ 353us) server_negotiation.cc:299] Negotiation successful
0928 02:57:14.630802 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":218,"threads_started":1}
I20260928 02:57:14.632678 15100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:14.620409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33877 (local address 127.27.229.193:52959)
0928 02:57:14.621527 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:14.621570 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:14.623490 (+ 1920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:14.624281 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:14.624291 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:14.624386 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:14.624995 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:14.625012 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.626070 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.626075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:14.627496 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:14.627510 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:14.628321 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:14.628328 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:14.628577 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:14.629502 (+ 925us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:14.629509 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:14.629594 (+ 85us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:14.629721 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:14.630400 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:14.630416 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:14.630428 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:57:14.630647 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 02:57:14.631160 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":309,"threads_started":1}
I20260928 02:57:14.634526 15097 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33877
I20260928 02:57:14.634958 15097 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:14.635809 15097 heartbeater.cc:511] Master 127.27.229.254:33877 requested a full tablet report, sending...
I20260928 02:57:14.638540 14911 ts_manager.cc:194] Registered new tserver with Master: 3e31928d96be472d810c438ea69d4ab2 (127.27.229.193:46733)
I20260928 02:57:14.640152 14911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.193:52959
W20260928 02:57:14.966296 15101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:14.966670 15101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:14.966873 15101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:14.974255 15101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:14.974403 15101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:14.974591 15101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:14.984529 15101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:14.985922 15101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:14.987725 15101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:15.001080 15106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:15.001134 15107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:15.001199 15109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:15.002259 15101 server_base.cc:1034] running on GCE node
I20260928 02:57:15.002954 15101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:15.004072 15101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:15.005303 15101 hybrid_clock.cc:648] HybridClock initialized: now 1790564235005262 us; error 59 us; skew 500 ppm
I20260928 02:57:15.005636 15101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:15.008280 15101 webserver.cc:460] Webserver started at http://127.27.229.194:37163/ using document root <none> and password file <none>
I20260928 02:57:15.009024 15101 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:15.009157 15101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:15.009480 15101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:15.012562 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "1807f2a494b7aee327761d6f16eeadf6"
server_key_iv: "92d3bb12d56e9889b3dca4d67fcc3bf2"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.013396 15101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "1807f2a494b7aee327761d6f16eeadf6"
server_key_iv: "92d3bb12d56e9889b3dca4d67fcc3bf2"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.018724 15101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:57:15.022403 15115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.024011 15101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:15.024204 15101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "1807f2a494b7aee327761d6f16eeadf6"
server_key_iv: "92d3bb12d56e9889b3dca4d67fcc3bf2"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.024585 15101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:15.040271 15101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:15.041177 15101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:15.041522 15101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:15.042578 15101 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:15.044281 15101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:15.044411 15101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.044571 15101 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:15.044672 15101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.079164 15101 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:37749
I20260928 02:57:15.079207 15227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:37749 every 8 connection(s)
I20260928 02:57:15.080611 15101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:15.090430 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15101
I20260928 02:57:15.090654 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:15.091418 28567 external_mini_cluster.cc:1428] Setting key 322dd88ebe9d84c90d5c37453cc487dc
I20260928 02:57:15.094223 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.083517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:51207 (local address 127.27.229.254:33877)
0928 02:57:15.083742 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.083748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.085402 (+ 1654us) server_negotiation.cc:407] Connection header received
0928 02:57:15.087143 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.087148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.087218 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.087316 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:15.088411 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.089118 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.090928 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.091687 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.091845 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.093414 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:15.093418 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:15.093421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:15.093437 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:15.093469 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:15.093480 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:15.093584 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.093833 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.093932 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260928 02:57:15.094485 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--builtin_ntp_servers=127.27.229.212:40873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:15.095472 15230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.084062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33877 (local address 127.27.229.194:51207)
0928 02:57:15.085172 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.085216 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.086811 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.087537 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.087544 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.087634 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:15.088225 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.088241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.089275 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.089279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.090564 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.090576 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.091915 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.091921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.092143 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.093025 (+ 882us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:15.093028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:15.093072 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:15.093159 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:15.093586 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:15.093600 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:15.093609 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.093823 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.094225 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":386,"threads_started":1}
I20260928 02:57:15.096850 15228 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33877
I20260928 02:57:15.097162 15228 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:15.097950 15228 heartbeater.cc:511] Master 127.27.229.254:33877 requested a full tablet report, sending...
I20260928 02:57:15.099431 14911 ts_manager.cc:194] Registered new tserver with Master: f1dfdf3d6d164d7e870c4be74a2b2f7d (127.27.229.194:37749)
I20260928 02:57:15.100244 14911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.194:51207
W20260928 02:57:15.438103 15231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:15.438416 15231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:15.438513 15231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:15.446033 15231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:15.446154 15231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:15.446264 15231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:15.455636 15231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:33877
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:15.457075 15231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:15.459528 15231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:15.473954 15236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:15.473999 15237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:15.474489 15239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:15.474802 15231 server_base.cc:1034] running on GCE node
I20260928 02:57:15.475498 15231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:15.476536 15231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:15.477768 15231 hybrid_clock.cc:648] HybridClock initialized: now 1790564235477666 us; error 70 us; skew 500 ppm
I20260928 02:57:15.478109 15231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:15.480641 15231 webserver.cc:460] Webserver started at http://127.27.229.195:46245/ using document root <none> and password file <none>
I20260928 02:57:15.481455 15231 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:15.481602 15231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:15.481972 15231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:15.485296 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "97f85d16189640f6b3d7db0d2abe47e0"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "07e82ce37b615126eaeccb45838433f3"
server_key_iv: "1989484690b146d9254a7aac8405910b"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.486186 15231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "97f85d16189640f6b3d7db0d2abe47e0"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "07e82ce37b615126eaeccb45838433f3"
server_key_iv: "1989484690b146d9254a7aac8405910b"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.492110 15231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260928 02:57:15.495889 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.497311 15231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:57:15.497505 15231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "97f85d16189640f6b3d7db0d2abe47e0"
format_stamp: "Formatted at 2026-09-28 02:57:15 on dist-test-slave-s1kg"
server_key: "07e82ce37b615126eaeccb45838433f3"
server_key_iv: "1989484690b146d9254a7aac8405910b"
server_key_version: "encryptionkey@0"
I20260928 02:57:15.497720 15231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:15.528095 15231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:15.529023 15231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:15.529289 15231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:15.530452 15231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:15.531968 15231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:15.532065 15231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.532152 15231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:15.532199 15231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:15.565864 15231 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:37139
I20260928 02:57:15.565882 15357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:37139 every 8 connection(s)
I20260928 02:57:15.567399 15231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:15.570917 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15231
I20260928 02:57:15.571110 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:15.571563 28567 external_mini_cluster.cc:1428] Setting key 2dc206c9514b7b0cc0c6e16fa9ae19d9
I20260928 02:57:15.580863 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.570066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:50327 (local address 127.27.229.254:33877)
0928 02:57:15.570253 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.570258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.571835 (+ 1577us) server_negotiation.cc:407] Connection header received
0928 02:57:15.573884 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.573889 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.573974 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.574085 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:15.575336 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.576136 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.577791 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.578350 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.578467 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.580053 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:15.580059 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:15.580062 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:15.580083 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:15.580116 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:15.580128 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:15.580211 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.580484 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.580572 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:57:15.582027 15360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.570584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:33877 (local address 127.27.229.195:50327)
0928 02:57:15.571642 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.571693 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.573618 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.574299 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.574312 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.574435 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:15.575162 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.575181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.576295 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.576299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.577645 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.577657 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.578590 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.578595 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.578813 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.579698 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:15.579700 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:15.579743 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:15.579828 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:15.580225 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:15.580233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:15.580239 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.580384 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.580729 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":364,"threads_started":1}
I20260928 02:57:15.583679 15358 heartbeater.cc:347] Connected to a master server at 127.27.229.254:33877
I20260928 02:57:15.583971 15358 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:15.584687 15358 heartbeater.cc:511] Master 127.27.229.254:33877 requested a full tablet report, sending...
I20260928 02:57:15.586404 14911 ts_manager.cc:194] Registered new tserver with Master: 97f85d16189640f6b3d7db0d2abe47e0 (127.27.229.195:37139)
I20260928 02:57:15.587318 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:15.587534 14911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.195:50327
I20260928 02:57:15.597947 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.591718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.27.229.254:33877)
0928 02:57:15.592018 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.592026 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.592336 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:57:15.592585 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.592589 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.592662 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.592769 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:15.593289 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.594001 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.595479 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.595953 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.596104 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.597295 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:15.597300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:15.597303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:15.597322 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:15.597355 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:15.597366 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:15.597494 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.597648 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.597696 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:57:15.602077 14911 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44896:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:15.608896 14911 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:15.643790 15097 heartbeater.cc:503] Master 127.27.229.254:33877 was elected leader, sending a full tablet report...
I20260928 02:57:15.644917 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.621794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53460 (local address 127.27.229.194:37749)
0928 02:57:15.634112 (+ 12318us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.634121 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.634820 (+ 699us) server_negotiation.cc:407] Connection header received
0928 02:57:15.635144 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.635151 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.635439 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.635603 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:15.636557 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.638170 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.641495 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643029 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.643189 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.643409 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.644371 (+ 962us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.644520 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12140,"thread_start_us":12329,"threads_started":1}
I20260928 02:57:15.644997 15370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.621553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37749 (local address 127.0.0.1:53460)
0928 02:57:15.634708 (+ 13155us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.634726 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.634943 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.635888 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.635892 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.635912 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:15.636370 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.636383 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.638338 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.638341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.641251 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.641264 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643534 (+ 2270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.643541 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.643783 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.643897 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.644071 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.644594 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13000,"mutex_wait_us":495,"thread_start_us":12025,"threads_started":1}
I20260928 02:57:15.646744 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.621310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:46733 (local address 127.0.0.1:38146)
0928 02:57:15.634096 (+ 12786us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.634159 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.634419 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.635664 (+ 1245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.635673 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.635706 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:15.636250 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.636265 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.638331 (+ 2066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.638336 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.641250 (+ 2914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.641264 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643582 (+ 2318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.643590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.643786 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.646190 (+ 2404us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.646264 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.646403 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12466,"mutex_wait_us":2579,"thread_start_us":13105,"threads_started":1}
I20260928 02:57:15.647385 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.621794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.27.229.193:46733)
0928 02:57:15.633072 (+ 11278us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.633087 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.634699 (+ 1612us) server_negotiation.cc:407] Connection header received
0928 02:57:15.634859 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.634870 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.635149 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.635340 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:15.636485 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.638143 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.641545 (+ 3402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643028 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.643217 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.643409 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.646958 (+ 3549us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.647137 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10991,"thread_start_us":11313,"threads_started":1}
I20260928 02:57:15.648586 15375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.636185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:37139 (local address 127.0.0.1:37294)
0928 02:57:15.640623 (+ 4438us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.640638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.640806 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.641409 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.641415 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.641441 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:15.641821 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.641830 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643871 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.643875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.645973 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.645989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.647840 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.647848 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.648054 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.648097 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.648196 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.648319 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4335,"thread_start_us":3513,"threads_started":1}
I20260928 02:57:15.648780 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.635159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.27.229.195:37139)
0928 02:57:15.640391 (+ 5232us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.640398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.640754 (+ 356us) server_negotiation.cc:407] Connection header received
0928 02:57:15.640993 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.641002 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.641175 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.641427 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:15.641998 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.643726 (+ 1728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.646173 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.647651 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.647843 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.648048 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.648253 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.648411 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5059,"thread_start_us":4903,"threads_started":1}
I20260928 02:57:15.649519 15163 tablet_service.cc:1467] Processing CreateTablet for tablet 137c70502b0b4278adf186f084246941 (DEFAULT_TABLE table=test-table [id=e6f07796296841b78b2b35a44c81820a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:15.651136 15163 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c70502b0b4278adf186f084246941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:15.651780 15032 tablet_service.cc:1467] Processing CreateTablet for tablet 137c70502b0b4278adf186f084246941 (DEFAULT_TABLE table=test-table [id=e6f07796296841b78b2b35a44c81820a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:15.653204 15032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c70502b0b4278adf186f084246941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:15.653038 15293 tablet_service.cc:1467] Processing CreateTablet for tablet 137c70502b0b4278adf186f084246941 (DEFAULT_TABLE table=test-table [id=e6f07796296841b78b2b35a44c81820a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:15.654388 15293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137c70502b0b4278adf186f084246941. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:15.663549 15376 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Bootstrap starting.
I20260928 02:57:15.667150 15376 tablet_bootstrap.cc:654] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:15.668606 15376 log.cc:824] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:15.671912 15377 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Bootstrap starting.
I20260928 02:57:15.676009 15379 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Bootstrap starting.
I20260928 02:57:15.676630 15377 tablet_bootstrap.cc:654] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:15.678141 15377 log.cc:824] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:15.678637 15376 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: No bootstrap required, opened a new log
I20260928 02:57:15.679046 15376 ts_tablet_manager.cc:1397] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260928 02:57:15.679826 15379 tablet_bootstrap.cc:654] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:15.681244 15379 log.cc:824] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:15.682092 15377 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: No bootstrap required, opened a new log
I20260928 02:57:15.682636 15377 ts_tablet_manager.cc:1397] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20260928 02:57:15.685297 15379 tablet_bootstrap.cc:492] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: No bootstrap required, opened a new log
I20260928 02:57:15.685619 15379 ts_tablet_manager.cc:1397] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260928 02:57:15.687564 15376 raft_consensus.cc:348] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.687975 15376 raft_consensus.cc:374] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:15.688072 15376 raft_consensus.cc:729] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1dfdf3d6d164d7e870c4be74a2b2f7d, State: Initialized, Role: FOLLOWER
I20260928 02:57:15.688697 15376 consensus_queue.cc:260] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.690493 15228 heartbeater.cc:503] Master 127.27.229.254:33877 was elected leader, sending a full tablet report...
I20260928 02:57:15.691219 15376 ts_tablet_manager.cc:1428] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Time spent starting tablet: real 0.012s user 0.010s sys 0.002s
I20260928 02:57:15.693549 15377 raft_consensus.cc:348] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.693953 15377 raft_consensus.cc:374] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:15.694084 15377 raft_consensus.cc:729] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e31928d96be472d810c438ea69d4ab2, State: Initialized, Role: FOLLOWER
I20260928 02:57:15.694746 15377 consensus_queue.cc:260] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.696661 15377 ts_tablet_manager.cc:1428] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260928 02:57:15.697496 15379 raft_consensus.cc:348] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.698010 15379 raft_consensus.cc:374] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:15.698127 15379 raft_consensus.cc:729] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97f85d16189640f6b3d7db0d2abe47e0, State: Initialized, Role: FOLLOWER
I20260928 02:57:15.698859 15379 consensus_queue.cc:260] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.699970 15358 heartbeater.cc:503] Master 127.27.229.254:33877 was elected leader, sending a full tablet report...
I20260928 02:57:15.700533 15379 ts_tablet_manager.cc:1428] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
W20260928 02:57:15.819207 15359 tablet.cc:2369] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:15.833066 15229 tablet.cc:2369] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:15.853461 15383 raft_consensus.cc:482] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:15.853744 15383 raft_consensus.cc:504] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.855542 15383 leader_election.cc:288] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f1dfdf3d6d164d7e870c4be74a2b2f7d (127.27.229.194:37749), 97f85d16189640f6b3d7db0d2abe47e0 (127.27.229.195:37139)
I20260928 02:57:15.865375 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.856172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:34319 (local address 127.27.229.195:37139)
0928 02:57:15.856386 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.856393 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.857069 (+ 676us) server_negotiation.cc:407] Connection header received
0928 02:57:15.857378 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.857387 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.857482 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.857590 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:15.858328 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.859555 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.862933 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.864230 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.864407 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.864480 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.864978 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.865115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:57:15.865592 15386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.856601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37749 (local address 127.27.229.193:60113)
0928 02:57:15.857301 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.857315 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.857434 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.857780 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.857786 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.857839 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:15.858265 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.858276 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.859722 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.859728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.862631 (+ 2903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.862651 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.864486 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.864493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.864741 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.864803 (+ 62us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.864906 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.865099 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":294,"spinlock_wait_cycles":1920,"thread_start_us":383,"threads_started":1}
I20260928 02:57:15.865675 15385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.856083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:37139 (local address 127.27.229.193:34319)
0928 02:57:15.856915 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:15.856934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:15.857191 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:15.857623 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:15.857627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:15.857647 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:15.858140 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.858158 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.859721 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.859728 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.862640 (+ 2912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:15.862651 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.864502 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.864508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:15.864733 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.864799 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 02:57:15.864906 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:57:15.865099 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"mutex_wait_us":261,"spinlock_wait_cycles":1920,"thread_start_us":478,"threads_started":1}
I20260928 02:57:15.865375 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.856030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:60113 (local address 127.27.229.194:37749)
0928 02:57:15.856226 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.856231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.857423 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 02:57:15.857544 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.857552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.857647 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.857749 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:15.858465 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.859543 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.862921 (+ 3378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.864204 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.864401 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.864480 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.864978 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.865115 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:57:15.866803 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c70502b0b4278adf186f084246941" candidate_uuid: "3e31928d96be472d810c438ea69d4ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f85d16189640f6b3d7db0d2abe47e0" is_pre_election: true
I20260928 02:57:15.866940 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c70502b0b4278adf186f084246941" candidate_uuid: "3e31928d96be472d810c438ea69d4ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" is_pre_election: true
I20260928 02:57:15.867411 15313 raft_consensus.cc:2454] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e31928d96be472d810c438ea69d4ab2 in term 0.
I20260928 02:57:15.867431 15183 raft_consensus.cc:2454] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3e31928d96be472d810c438ea69d4ab2 in term 0.
I20260928 02:57:15.868139 14985 leader_election.cc:302] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3e31928d96be472d810c438ea69d4ab2, 97f85d16189640f6b3d7db0d2abe47e0, f1dfdf3d6d164d7e870c4be74a2b2f7d; no voters:
I20260928 02:57:15.868606 15383 raft_consensus.cc:2781] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:15.868861 15383 raft_consensus.cc:482] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:15.869024 15383 raft_consensus.cc:3037] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:57:15.869225 15098 tablet.cc:2369] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:15.871447 15383 raft_consensus.cc:504] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.872308 15383 leader_election.cc:288] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [CANDIDATE]: Term 1 election: Requested vote from peers f1dfdf3d6d164d7e870c4be74a2b2f7d (127.27.229.194:37749), 97f85d16189640f6b3d7db0d2abe47e0 (127.27.229.195:37139)
I20260928 02:57:15.872977 15183 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c70502b0b4278adf186f084246941" candidate_uuid: "3e31928d96be472d810c438ea69d4ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d"
I20260928 02:57:15.873095 15313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "137c70502b0b4278adf186f084246941" candidate_uuid: "3e31928d96be472d810c438ea69d4ab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97f85d16189640f6b3d7db0d2abe47e0"
I20260928 02:57:15.873196 15183 raft_consensus.cc:3037] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:15.873396 15313 raft_consensus.cc:3037] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:15.875586 15183 raft_consensus.cc:2454] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e31928d96be472d810c438ea69d4ab2 in term 1.
I20260928 02:57:15.875970 15313 raft_consensus.cc:2454] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3e31928d96be472d810c438ea69d4ab2 in term 1.
I20260928 02:57:15.876047 14987 leader_election.cc:302] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e31928d96be472d810c438ea69d4ab2, f1dfdf3d6d164d7e870c4be74a2b2f7d; no voters:
I20260928 02:57:15.876523 15383 raft_consensus.cc:2781] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:15.877169 15383 raft_consensus.cc:686] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 1 LEADER]: Becoming Leader. State: Replica: 3e31928d96be472d810c438ea69d4ab2, State: Running, Role: LEADER
I20260928 02:57:15.877743 15383 consensus_queue.cc:237] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } }
I20260928 02:57:15.882787 14910 catalog_manager.cc:5495] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3e31928d96be472d810c438ea69d4ab2 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "3e31928d96be472d810c438ea69d4ab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e31928d96be472d810c438ea69d4ab2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 46733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 } health_report { overall_health: UNKNOWN } } }
I20260928 02:57:15.949222 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.942894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.27.229.193:46733)
0928 02:57:15.943125 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.943132 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:15.943214 (+ 82us) server_negotiation.cc:407] Connection header received
0928 02:57:15.943501 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:15.943506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:15.943600 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:15.943761 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:15.944167 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.945331 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.946794 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:15.947308 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:15.947461 (+ 153us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:15.948063 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:15.948587 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:15.948774 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:15.948942 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 02:57:15.948987 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:57:15.957857 15183 raft_consensus.cc:1261] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 1 FOLLOWER]: Refusing update from remote peer 3e31928d96be472d810c438ea69d4ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:15.957856 15313 raft_consensus.cc:1261] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 1 FOLLOWER]: Refusing update from remote peer 3e31928d96be472d810c438ea69d4ab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:15.958899 15383 consensus_queue.cc:1035] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1dfdf3d6d164d7e870c4be74a2b2f7d" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:15.959214 15387 consensus_queue.cc:1035] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97f85d16189640f6b3d7db0d2abe47e0" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 37139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:15.971962 15390 mvcc.cc:205] Tried to move back new op lower bound from 7334151110473273344 to 7334151110159728640. Current Snapshot: MvccSnapshot[applied={T|T < 7334151110473273344}]
I20260928 02:57:15.986172 14911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:15.986521 14911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44896:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:15.991014 14911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P daa7f6dcd54c4b89b0ae075d20f0b1ac: Sending DeleteTablet for 3 replicas of tablet 137c70502b0b4278adf186f084246941
I20260928 02:57:15.992488 15032 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c70502b0b4278adf186f084246941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:15 UTC) from {username='slave'} at 127.0.0.1:38146
I20260928 02:57:15.992537 15163 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c70502b0b4278adf186f084246941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:15 UTC) from {username='slave'} at 127.0.0.1:53460
I20260928 02:57:15.992676 15293 tablet_service.cc:1514] Processing DeleteTablet for tablet 137c70502b0b4278adf186f084246941 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:15 UTC) from {username='slave'} at 127.0.0.1:37294
I20260928 02:57:15.994154 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14970
I20260928 02:57:16.003681 15399 tablet_replica.cc:330] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: stopping tablet replica
I20260928 02:57:16.004330 15401 tablet_replica.cc:330] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: stopping tablet replica
I20260928 02:57:16.004235 15399 raft_consensus.cc:2229] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:16.004851 15399 raft_consensus.cc:2258] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:16.004920 15401 raft_consensus.cc:2229] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:16.005584 15401 raft_consensus.cc:2258] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:16.006541 15400 tablet_replica.cc:330] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: stopping tablet replica
I20260928 02:57:16.007035 15400 raft_consensus.cc:2229] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:16.007807 15400 raft_consensus.cc:2258] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:16.008105 15401 ts_tablet_manager.cc:1905] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:16.010107 15400 ts_tablet_manager.cc:1905] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:16.016254 15401 ts_tablet_manager.cc:1918] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:16.016461 15401 log.cc:1196] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/137c70502b0b4278adf186f084246941
I20260928 02:57:16.016983 15401 ts_tablet_manager.cc:1939] T 137c70502b0b4278adf186f084246941 P 97f85d16189640f6b3d7db0d2abe47e0: Deleting consensus metadata
I20260928 02:57:16.017596 15399 ts_tablet_manager.cc:1905] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:16.018328 15400 ts_tablet_manager.cc:1918] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:16.018589 15400 log.cc:1196] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/137c70502b0b4278adf186f084246941
I20260928 02:57:16.019176 14895 catalog_manager.cc:4841] TS 97f85d16189640f6b3d7db0d2abe47e0 (127.27.229.195:37139): tablet 137c70502b0b4278adf186f084246941 (table test-table [id=e6f07796296841b78b2b35a44c81820a]) successfully deleted
I20260928 02:57:16.019972 15400 ts_tablet_manager.cc:1939] T 137c70502b0b4278adf186f084246941 P 3e31928d96be472d810c438ea69d4ab2: Deleting consensus metadata
I20260928 02:57:16.021644 14895 catalog_manager.cc:4841] TS 3e31928d96be472d810c438ea69d4ab2 (127.27.229.193:46733): tablet 137c70502b0b4278adf186f084246941 (table test-table [id=e6f07796296841b78b2b35a44c81820a]) successfully deleted
I20260928 02:57:16.023986 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:15.995180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.27.229.193:46733)
0928 02:57:15.995368 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:15.995376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:16.009125 (+ 13749us) server_negotiation.cc:407] Connection header received
0928 02:57:16.009672 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:16.009678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:16.009787 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:16.009869 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:16.010651 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.011941 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.013898 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.014584 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.014695 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:16.022336 (+ 7641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:16.022346 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:16.022364 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:16.023116 (+ 752us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:16.023181 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:16.023206 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:16.023344 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:16.023505 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 02:57:16.023618 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:57:16.024863 15092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:16.024927 15399 ts_tablet_manager.cc:1918] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:16.025094 15399 log.cc:1196] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/137c70502b0b4278adf186f084246941
I20260928 02:57:16.025662 15399 ts_tablet_manager.cc:1939] T 137c70502b0b4278adf186f084246941 P f1dfdf3d6d164d7e870c4be74a2b2f7d: Deleting consensus metadata
I20260928 02:57:16.027993 14897 catalog_manager.cc:4841] TS f1dfdf3d6d164d7e870c4be74a2b2f7d (127.27.229.194:37749): tablet 137c70502b0b4278adf186f084246941 (table test-table [id=e6f07796296841b78b2b35a44c81820a]) successfully deleted
I20260928 02:57:16.125953 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14970
I20260928 02:57:16.144935 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15101
I20260928 02:57:16.151975 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:16.145786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.27.229.194:37749)
0928 02:57:16.145972 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:16.145978 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:16.146000 (+ 22us) server_negotiation.cc:407] Connection header received
0928 02:57:16.146142 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:16.146146 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:16.146240 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:16.146362 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:16.146757 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.147583 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.148866 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.149277 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.149436 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:16.150541 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:16.150550 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:16.150564 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:16.151212 (+ 648us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:16.151259 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:16.151279 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:16.151410 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:16.151536 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:57:16.151642 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 02:57:16.152742 15223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:16.242326 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15101
I20260928 02:57:16.259249 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15231
I20260928 02:57:16.266541 15374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:16.260117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37300 (local address 127.27.229.195:37139)
0928 02:57:16.260282 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:16.260287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:16.260305 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:57:16.260386 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:16.260390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:16.260509 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:16.260604 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:16.261039 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.261882 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.263324 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:16.263720 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:16.263818 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:16.264949 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:16.264958 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:16.264972 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:16.265756 (+ 784us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:16.265826 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:16.265848 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:16.265986 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:16.266120 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 02:57:16.266227 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:57:16.267283 15353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:16.356701 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15231
I20260928 02:57:16.373994 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 14880
I20260928 02:57:16.375257 14941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:16.486752 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 14880
2026-09-28T02:57:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2986 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:16 dist-test-slave-s1kg krb5kdc[15409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:16 dist-test-slave-s1kg krb5kdc[15409](info): set up 2 sockets
Sep 28 02:57:16 dist-test-slave-s1kg krb5kdc[15409](info): commencing operation
krb5kdc: starting...
W20260928 02:57:17.611225 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:17 dist-test-slave-s1kg krb5kdc[15409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:17.874820 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37821
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:37821
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:18.218605 15422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:18.219146 15422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:18.219316 15422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:18.226732 15422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:18.226943 15422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:18.227056 15422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:18.227161 15422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:18.227257 15422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:18.237581 15422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:37821
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37821
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:18.239032 15422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:18.240356 15422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:18.252703 15427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:18.252775 15428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:18.252738 15430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:18.253654 15422 server_base.cc:1034] running on GCE node
I20260928 02:57:18.254477 15422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:18.256047 15422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:18.257270 15422 hybrid_clock.cc:648] HybridClock initialized: now 1790564238257219 us; error 43 us; skew 500 ppm
I20260928 02:57:18.257643 15422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:18 dist-test-slave-s1kg krb5kdc[15409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564238, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:18.261071 15422 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:18.262732 15422 webserver.cc:460] Webserver started at http://127.27.229.254:37117/ using document root <none> and password file <none>
I20260928 02:57:18.263504 15422 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:18.263670 15422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:18.263978 15422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:18.267236 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "25896212f3e44e95a37468d70759793c"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "dfed4b7e0e44a4a70bbf4e0be652fa2a"
server_key_iv: "3f61c5f049ec60ef9d7a0010e870c1fe"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.268126 15422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "25896212f3e44e95a37468d70759793c"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "dfed4b7e0e44a4a70bbf4e0be652fa2a"
server_key_iv: "3f61c5f049ec60ef9d7a0010e870c1fe"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.273552 15422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:57:18.277354 15437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:18.278734 15422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 02:57:18.278986 15422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "25896212f3e44e95a37468d70759793c"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "dfed4b7e0e44a4a70bbf4e0be652fa2a"
server_key_iv: "3f61c5f049ec60ef9d7a0010e870c1fe"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.279203 15422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:18.303004 15422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:18.307972 15422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:18.308296 15422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:18.328637 15422 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:37821
I20260928 02:57:18.328675 15488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:37821 every 8 connection(s)
I20260928 02:57:18.330049 15422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:18.330740 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15422
I20260928 02:57:18.330973 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:18.331554 28567 external_mini_cluster.cc:1428] Setting key f5c76154246e8e8d21956421cc78d000
I20260928 02:57:18.334887 15489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:57:18 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:18.345970 15489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: Bootstrap starting.
I20260928 02:57:18.349287 15489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:18.350383 15489 log.cc:824] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:18.353471 15489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: No bootstrap required, opened a new log
I20260928 02:57:18.354408 15492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:18.333495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36726 (local address 127.27.229.254:37821)
0928 02:57:18.334286 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:18.334298 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:18.334338 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:57:18.336234 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:18.336256 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:18.336631 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:18.337033 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:18.338597 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.340070 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.341362 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.342047 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.342253 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:18.346659 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:18.346667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:18.346683 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:18.346721 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:18.350029 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:18.350872 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:18.350882 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:18.350896 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:18.351105 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:18.351562 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:18.351569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:18.351576 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:18.352132 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:18.352444 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:18.352909 (+ 465us) server_negotiation.cc:299] Negotiation successful
0928 02:57:18.353362 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":362,"threads_started":1}
I20260928 02:57:18.362129 15489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } }
I20260928 02:57:18.362663 15489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:18.362793 15489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25896212f3e44e95a37468d70759793c, State: Initialized, Role: FOLLOWER
I20260928 02:57:18.363449 15489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } }
I20260928 02:57:18.363713 15489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:18.363859 15489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:18.364037 15489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:18.366313 15489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } }
I20260928 02:57:18.366986 15489 leader_election.cc:302] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 25896212f3e44e95a37468d70759793c; no voters:
I20260928 02:57:18.367739 15489 leader_election.cc:288] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:18.367861 15494 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:18.368489 15494 raft_consensus.cc:686] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [term 1 LEADER]: Becoming Leader. State: Replica: 25896212f3e44e95a37468d70759793c, State: Running, Role: LEADER
I20260928 02:57:18.369082 15494 consensus_queue.cc:237] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } }
I20260928 02:57:18.370079 15489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:18.371887 15495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "25896212f3e44e95a37468d70759793c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } } }
I20260928 02:57:18.372306 15495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:18.372324 15496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 25896212f3e44e95a37468d70759793c. Latest consensus state: current_term: 1 leader_uuid: "25896212f3e44e95a37468d70759793c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "25896212f3e44e95a37468d70759793c" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37821 } } }
I20260928 02:57:18.373564 15496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:18.373816 15499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:18.379587 15499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:18.390861 15499 catalog_manager.cc:1347] Generated new cluster ID: 04cd5196ae8f498eb8480212b5aff822
I20260928 02:57:18.391004 15499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:18.408017 15499 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:18.409415 15499 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:18.416884 15499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: Generated new TSK 0
I20260928 02:57:18.417912 15499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.229.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
Entry for principal kudu/127.27.229.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab.
I20260928 02:57:18.480229 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:18.850183 15515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:18.850478 15515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:18.850642 15515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:18.858435 15515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:18.858566 15515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:18.858749 15515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:18.869114 15515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.193.keytab
--principal=kudu/127.27.229.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:18.870637 15515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:18.872128 15515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:18.887081 15520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:18.887450 15523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:18.887585 15521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:18.888852 15515 server_base.cc:1034] running on GCE node
I20260928 02:57:18.889679 15515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:18.890786 15515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:18.892045 15515 hybrid_clock.cc:648] HybridClock initialized: now 1790564238891979 us; error 60 us; skew 500 ppm
I20260928 02:57:18.892374 15515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:18 dist-test-slave-s1kg krb5kdc[15409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564238, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:18.895877 15515 init.cc:377] Logged in from keytab as kudu/127.27.229.193@KRBTEST.COM (short username kudu)
I20260928 02:57:18.897557 15515 webserver.cc:460] Webserver started at http://127.27.229.193:45725/ using document root <none> and password file <none>
I20260928 02:57:18.898288 15515 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:18.898468 15515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:18.898797 15515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:18.901863 15515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "49401898d8fa4a428392a57cf1c997f2"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "bf843239940cdf2240d8fce60bc9b037"
server_key_iv: "688386ae68b9920d9637eefb5554ddcc"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.902685 15515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "49401898d8fa4a428392a57cf1c997f2"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "bf843239940cdf2240d8fce60bc9b037"
server_key_iv: "688386ae68b9920d9637eefb5554ddcc"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.908166 15515 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:57:18.911950 15530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:18.913348 15515 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:18.913546 15515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "49401898d8fa4a428392a57cf1c997f2"
format_stamp: "Formatted at 2026-09-28 02:57:18 on dist-test-slave-s1kg"
server_key: "bf843239940cdf2240d8fce60bc9b037"
server_key_iv: "688386ae68b9920d9637eefb5554ddcc"
server_key_version: "encryptionkey@0"
I20260928 02:57:18.913772 15515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:18.931876 15515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:18.936709 15515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:18.936971 15515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:18.937929 15515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:18.939383 15515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:18.939481 15515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:18.939574 15515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:18.939616 15515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:18.973778 15515 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:38637
I20260928 02:57:18.973835 15642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:38637 every 8 connection(s)
I20260928 02:57:18.975111 15515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:18.982333 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15515
I20260928 02:57:18.982566 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:18.983098 28567 external_mini_cluster.cc:1428] Setting key 95ae1813be26f5086af2d6cc21e39a1d
Sep 28 02:57:18 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564238, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.193@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:18.996547 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:18.977928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:46907 (local address 127.27.229.254:37821)
0928 02:57:18.978469 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:18.978476 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:18.979450 (+ 974us) server_negotiation.cc:407] Connection header received
0928 02:57:18.981283 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:18.981288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:18.981357 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:18.981460 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:18.983326 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.984245 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.986042 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.986460 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.986602 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:18.990182 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:18.990187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:18.990190 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:18.990222 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:18.993379 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:18.994130 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:18.994136 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:18.994140 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:18.994270 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:18.994715 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:18.994724 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:18.994726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:18.995053 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:18.995209 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:18.995966 (+ 757us) server_negotiation.cc:299] Negotiation successful
0928 02:57:18.996194 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":253,"threads_started":1}
I20260928 02:57:18.997625 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:18.978501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37821 (local address 127.27.229.193:46907)
0928 02:57:18.979286 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:18.979327 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:18.980985 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:18.981735 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:18.981744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:18.982335 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:18.983124 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:18.983145 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.984470 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.984474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:18.985881 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:18.985894 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:18.986705 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:18.986713 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:18.987021 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:18.987732 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:18.987759 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:18.989933 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:18.993526 (+ 3593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:18.993536 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:18.993554 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:18.993994 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:18.994390 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:18.994395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:18.994399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:18.994591 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:18.995181 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:18.995189 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:18.995558 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 02:57:18.995831 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 02:57:18.996383 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":358,"threads_started":1}
I20260928 02:57:18.999493 15643 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37821
I20260928 02:57:18.999817 15643 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:19.000581 15643 heartbeater.cc:511] Master 127.27.229.254:37821 requested a full tablet report, sending...
I20260928 02:57:19.003027 15454 ts_manager.cc:194] Registered new tserver with Master: 49401898d8fa4a428392a57cf1c997f2 (127.27.229.193:38637)
I20260928 02:57:19.005402 15454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.193@KRBTEST.COM'} at 127.27.229.193:46907
WARNING: no policy specified for kudu/127.27.229.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
Entry for principal kudu/127.27.229.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab.
I20260928 02:57:19.043097 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:19.390125 15649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:19.390489 15649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:19.390604 15649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:19.398376 15649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:19.398559 15649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:19.398813 15649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:19.413632 15649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.194.keytab
--principal=kudu/127.27.229.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:19.415060 15649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:19.416358 15649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:19.430575 15654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:19.430608 15655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:19.430527 15657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:19.431671 15649 server_base.cc:1034] running on GCE node
I20260928 02:57:19.432391 15649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:19.433591 15649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:19.434914 15649 hybrid_clock.cc:648] HybridClock initialized: now 1790564239434785 us; error 63 us; skew 500 ppm
I20260928 02:57:19.435280 15649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:19 dist-test-slave-s1kg krb5kdc[15409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564239, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:19.438593 15649 init.cc:377] Logged in from keytab as kudu/127.27.229.194@KRBTEST.COM (short username kudu)
I20260928 02:57:19.440302 15649 webserver.cc:460] Webserver started at http://127.27.229.194:38213/ using document root <none> and password file <none>
I20260928 02:57:19.441032 15649 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:19.441184 15649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:19.441517 15649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:19.444653 15649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a"
format_stamp: "Formatted at 2026-09-28 02:57:19 on dist-test-slave-s1kg"
server_key: "5f44571b90342e8ce95790db39bf45fa"
server_key_iv: "9c2b33754cad1c835b2409e86f5b0dc0"
server_key_version: "encryptionkey@0"
I20260928 02:57:19.445470 15649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a"
format_stamp: "Formatted at 2026-09-28 02:57:19 on dist-test-slave-s1kg"
server_key: "5f44571b90342e8ce95790db39bf45fa"
server_key_iv: "9c2b33754cad1c835b2409e86f5b0dc0"
server_key_version: "encryptionkey@0"
I20260928 02:57:19.451063 15649 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:57:19.454519 15664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:19.455888 15649 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:19.456079 15649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a"
format_stamp: "Formatted at 2026-09-28 02:57:19 on dist-test-slave-s1kg"
server_key: "5f44571b90342e8ce95790db39bf45fa"
server_key_iv: "9c2b33754cad1c835b2409e86f5b0dc0"
server_key_version: "encryptionkey@0"
I20260928 02:57:19.456290 15649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:19.471063 15649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:19.476742 15649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:19.477054 15649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:19.478087 15649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:19.479555 15649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:19.479660 15649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:19.479808 15649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:19.479885 15649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:19.515421 15649 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:37637
I20260928 02:57:19.515513 15776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:37637 every 8 connection(s)
I20260928 02:57:19.516820 15649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:19.519531 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15649
I20260928 02:57:19.519733 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:19.520155 28567 external_mini_cluster.cc:1428] Setting key 756e7d31ba1e04a6c37dbaf113956fd0
Sep 28 02:57:19 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564239, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.194@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:19.540145 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:19.519672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:37769 (local address 127.27.229.254:37821)
0928 02:57:19.521013 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:19.521020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:19.521686 (+ 666us) server_negotiation.cc:407] Connection header received
0928 02:57:19.523727 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:19.523732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:19.523826 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:19.523958 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:19.525531 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:19.526259 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:19.527895 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:19.528334 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:19.528479 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:19.532191 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:19.532200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:19.532208 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:19.532257 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:19.536180 (+ 3923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:19.537197 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:19.537205 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:19.537211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:19.537370 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:19.537927 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:19.537935 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:19.537940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:19.538349 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:19.538508 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:19.539369 (+ 861us) server_negotiation.cc:299] Negotiation successful
0928 02:57:19.539634 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1094,"threads_started":1}
I20260928 02:57:19.540797 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:19.520413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37821 (local address 127.27.229.194:37769)
0928 02:57:19.521534 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:19.521575 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:19.523447 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:19.524200 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:19.524208 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:19.524764 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:19.525339 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:19.525356 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:19.526425 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:19.526429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:19.527743 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:19.527755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:19.528573 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:19.528580 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:19.528812 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:19.529564 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:19.529588 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:19.531918 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:19.536497 (+ 4579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:19.536506 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:19.536522 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:19.537038 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:19.537555 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:19.537559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:19.537562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:19.537783 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:19.538480 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:19.538487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:19.538948 (+ 461us) client_negotiation.cc:769] Sending connection context
0928 02:57:19.539202 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 02:57:19.539646 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":611,"threads_started":1}
I20260928 02:57:19.542371 15777 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37821
I20260928 02:57:19.542738 15777 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:19.543689 15777 heartbeater.cc:511] Master 127.27.229.254:37821 requested a full tablet report, sending...
I20260928 02:57:19.545650 15454 ts_manager.cc:194] Registered new tserver with Master: 4f35ad04a4ff4b2bb1ad4fcc524ac13a (127.27.229.194:37637)
I20260928 02:57:19.546568 15454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.194@KRBTEST.COM'} at 127.27.229.194:37769
WARNING: no policy specified for kudu/127.27.229.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
Entry for principal kudu/127.27.229.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab.
I20260928 02:57:19.581671 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:33277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:19.967756 15783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:19.968075 15783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:19.968250 15783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:19.976653 15783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:19.976841 15783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:19.977041 15783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:19.987785 15783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:33277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.195.keytab
--principal=kudu/127.27.229.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:37821
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:19.989192 15783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:19.990720 15783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:20.006261 15791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:20.006289 15788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:20.006336 15783 server_base.cc:1034] running on GCE node
W20260928 02:57:20.006220 15789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:20.007479 15783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:20.008569 15783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:20.009019 15643 heartbeater.cc:503] Master 127.27.229.254:37821 was elected leader, sending a full tablet report...
I20260928 02:57:20.009814 15783 hybrid_clock.cc:648] HybridClock initialized: now 1790564240009776 us; error 44 us; skew 500 ppm
I20260928 02:57:20.010182 15783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:20 dist-test-slave-s1kg krb5kdc[15409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564240, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:20.013711 15783 init.cc:377] Logged in from keytab as kudu/127.27.229.195@KRBTEST.COM (short username kudu)
I20260928 02:57:20.015488 15783 webserver.cc:460] Webserver started at http://127.27.229.195:40655/ using document root <none> and password file <none>
I20260928 02:57:20.016208 15783 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:20.016315 15783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:20.016584 15783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:20.020012 15783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1"
format_stamp: "Formatted at 2026-09-28 02:57:20 on dist-test-slave-s1kg"
server_key: "f8923ce64c67c294c341084cae82c045"
server_key_iv: "9c50e128ce6df74027c4124fc4e2199d"
server_key_version: "encryptionkey@0"
I20260928 02:57:20.020864 15783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1"
format_stamp: "Formatted at 2026-09-28 02:57:20 on dist-test-slave-s1kg"
server_key: "f8923ce64c67c294c341084cae82c045"
server_key_iv: "9c50e128ce6df74027c4124fc4e2199d"
server_key_version: "encryptionkey@0"
I20260928 02:57:20.026351 15783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:57:20.030066 15798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:20.031690 15783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:20.031876 15783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1"
format_stamp: "Formatted at 2026-09-28 02:57:20 on dist-test-slave-s1kg"
server_key: "f8923ce64c67c294c341084cae82c045"
server_key_iv: "9c50e128ce6df74027c4124fc4e2199d"
server_key_version: "encryptionkey@0"
I20260928 02:57:20.032055 15783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:20.066054 15783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:20.070811 15783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:20.071087 15783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:20.072327 15783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:20.073695 15783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:20.073770 15783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:20.073871 15783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:20.073915 15783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:20.111246 15783 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:40319
I20260928 02:57:20.111353 15910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:40319 every 8 connection(s)
I20260928 02:57:20.112659 15783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:20.122972 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15783
I20260928 02:57:20.123183 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:20.123682 28567 external_mini_cluster.cc:1428] Setting key d2b816cc664de8bee96b226684a8ea6f
Sep 28 02:57:20 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564240, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.195@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:20.136919 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.115688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:48935 (local address 127.27.229.254:37821)
0928 02:57:20.116218 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.116226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.117356 (+ 1130us) server_negotiation.cc:407] Connection header received
0928 02:57:20.119412 (+ 2056us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.119418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.119508 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.119618 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:20.122064 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.123152 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.125464 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.126107 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.126257 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.130044 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:20.130049 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:20.130052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:20.130088 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:20.132907 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.134204 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.134208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.134210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.134358 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.134940 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.134944 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.134946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.135260 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:20.135441 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.136324 (+ 883us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.136560 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":249,"threads_started":1}
I20260928 02:57:20.138020 15914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.116355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:37821 (local address 127.27.229.195:48935)
0928 02:57:20.117160 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.117216 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.119107 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.119894 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.119903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.120585 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:20.121719 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.121750 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.123401 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.123407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.125281 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.125298 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.126408 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.126414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.126731 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.127530 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:57:20.127558 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:20.129800 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:20.133186 (+ 3386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:20.133202 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:20.133223 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:20.134049 (+ 826us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:20.134498 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:57:20.134502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:57:20.134505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:57:20.134732 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:57:20.135437 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:20.135443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:20.135860 (+ 417us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.136150 (+ 290us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.136634 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":346,"threads_started":1}
I20260928 02:57:20.139734 15911 heartbeater.cc:347] Connected to a master server at 127.27.229.254:37821
I20260928 02:57:20.140046 15911 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:20.140720 15911 heartbeater.cc:511] Master 127.27.229.254:37821 requested a full tablet report, sending...
I20260928 02:57:20.142161 15454 ts_manager.cc:194] Registered new tserver with Master: e8a92338991d4b6f8eef6cc6cd54b1b1 (127.27.229.195:40319)
I20260928 02:57:20.143112 15454 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.229.195@KRBTEST.COM'} at 127.27.229.195:48935
I20260928 02:57:20.153491 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:20.171049 15913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.157895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36740 (local address 127.27.229.254:37821)
0928 02:57:20.158179 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.158186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.158483 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:57:20.158642 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.158647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.158729 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.158819 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:20.160552 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.161279 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.162598 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.163051 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.163186 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.165598 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:20.165603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:20.165606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:20.165642 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:20.168289 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.168965 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.168971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.168974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.169102 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.169475 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.169487 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.169491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.169819 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:20.170081 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.170463 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.170713 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260928 02:57:20.175568 15454 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:20.182809 15454 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:20.208861 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.197711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36288 (local address 127.27.229.194:37637)
0928 02:57:20.198472 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.198480 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.198512 (+ 32us) server_negotiation.cc:407] Connection header received
0928 02:57:20.198911 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.198919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.199197 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.199409 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:20.201154 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.202992 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.205990 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.207477 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.207672 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.207874 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.208301 (+ 427us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.208493 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":485,"threads_started":1}
I20260928 02:57:20.210011 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.197588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37637 (local address 127.0.0.1:36288)
0928 02:57:20.198397 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.198436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.198625 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.199485 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.199495 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.199529 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:20.200974 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.200987 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.203134 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.203139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.205777 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.205793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.207680 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.207686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.207944 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.208043 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.209363 (+ 1320us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.209533 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"mutex_wait_us":4,"thread_start_us":936,"threads_started":1}
I20260928 02:57:20.210297 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.199743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35598 (local address 127.27.229.195:40319)
0928 02:57:20.201507 (+ 1764us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.201517 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.201556 (+ 39us) server_negotiation.cc:407] Connection header received
0928 02:57:20.201660 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.201669 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.201935 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.202150 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:20.202884 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.204398 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.207211 (+ 2813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.208694 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.208880 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.209184 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.209740 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.209942 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1565,"thread_start_us":1424,"threads_started":1}
I20260928 02:57:20.210335 15926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.199432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:40319 (local address 127.0.0.1:35598)
0928 02:57:20.200598 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.200619 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.200734 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.202132 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.202136 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.202163 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:20.202591 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.202605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.204553 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.204560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.207037 (+ 2477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.207053 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.209322 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.209328 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.209526 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.209559 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.209665 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.209815 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1042,"mutex_wait_us":88,"thread_start_us":844,"threads_started":1}
I20260928 02:57:20.211503 15929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.201277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38637 (local address 127.0.0.1:45310)
0928 02:57:20.201778 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.201796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.201908 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.203486 (+ 1578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.203490 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.203517 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:20.204586 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.204600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.205952 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.205957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.207947 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.207964 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.210631 (+ 2667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.210641 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.210818 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.210881 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.210988 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.211124 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":116,"thread_start_us":265,"threads_started":1}
I20260928 02:57:20.211596 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.200962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45310 (local address 127.27.229.193:38637)
0928 02:57:20.202919 (+ 1957us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.202925 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.202946 (+ 21us) server_negotiation.cc:407] Connection header received
0928 02:57:20.203019 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.203024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.203267 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.203986 (+ 719us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:20.204793 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.205788 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.208154 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.209685 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.209861 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.210027 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.211060 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.211239 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1730,"thread_start_us":1534,"threads_started":1}
I20260928 02:57:20.214552 15712 tablet_service.cc:1467] Processing CreateTablet for tablet 25dca14d4734424f8e7a26a49498a13c (DEFAULT_TABLE table=test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:20.215011 15578 tablet_service.cc:1467] Processing CreateTablet for tablet 25dca14d4734424f8e7a26a49498a13c (DEFAULT_TABLE table=test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:20.215058 15846 tablet_service.cc:1467] Processing CreateTablet for tablet 25dca14d4734424f8e7a26a49498a13c (DEFAULT_TABLE table=test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:20.216158 15712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dca14d4734424f8e7a26a49498a13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:20.216351 15578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dca14d4734424f8e7a26a49498a13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:20.216569 15846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25dca14d4734424f8e7a26a49498a13c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:20.231439 15930 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Bootstrap starting.
I20260928 02:57:20.232895 15932 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Bootstrap starting.
I20260928 02:57:20.233732 15931 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Bootstrap starting.
I20260928 02:57:20.236253 15930 tablet_bootstrap.cc:654] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:20.237144 15932 tablet_bootstrap.cc:654] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:20.237562 15931 tablet_bootstrap.cc:654] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:20.237634 15930 log.cc:824] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:20.238300 15932 log.cc:824] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:20.239352 15931 log.cc:824] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:20.242725 15932 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: No bootstrap required, opened a new log
I20260928 02:57:20.242962 15930 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: No bootstrap required, opened a new log
I20260928 02:57:20.242985 15931 tablet_bootstrap.cc:492] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: No bootstrap required, opened a new log
I20260928 02:57:20.243121 15932 ts_tablet_manager.cc:1397] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260928 02:57:20.243242 15931 ts_tablet_manager.cc:1397] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260928 02:57:20.243243 15930 ts_tablet_manager.cc:1397] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260928 02:57:20.250967 15932 raft_consensus.cc:348] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.251433 15932 raft_consensus.cc:374] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:20.251529 15932 raft_consensus.cc:729] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f35ad04a4ff4b2bb1ad4fcc524ac13a, State: Initialized, Role: FOLLOWER
I20260928 02:57:20.252189 15932 consensus_queue.cc:260] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.254200 15931 raft_consensus.cc:348] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.254199 15930 raft_consensus.cc:348] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.254657 15931 raft_consensus.cc:374] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:20.254657 15930 raft_consensus.cc:374] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:20.254767 15930 raft_consensus.cc:729] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8a92338991d4b6f8eef6cc6cd54b1b1, State: Initialized, Role: FOLLOWER
I20260928 02:57:20.254766 15931 raft_consensus.cc:729] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49401898d8fa4a428392a57cf1c997f2, State: Initialized, Role: FOLLOWER
I20260928 02:57:20.255565 15777 heartbeater.cc:503] Master 127.27.229.254:37821 was elected leader, sending a full tablet report...
I20260928 02:57:20.255558 15930 consensus_queue.cc:260] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.255559 15931 consensus_queue.cc:260] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.256347 15932 ts_tablet_manager.cc:1428] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260928 02:57:20.258606 15911 heartbeater.cc:503] Master 127.27.229.254:37821 was elected leader, sending a full tablet report...
I20260928 02:57:20.259202 15930 ts_tablet_manager.cc:1428] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Time spent starting tablet: real 0.016s user 0.015s sys 0.001s
I20260928 02:57:20.260126 15931 ts_tablet_manager.cc:1428] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
W20260928 02:57:20.269138 15778 tablet.cc:2369] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:20.364576 15912 tablet.cc:2369] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 02:57:20.477819 15644 tablet.cc:2369] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:57:20.565770 15937 raft_consensus.cc:482] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:20.566181 15937 raft_consensus.cc:504] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.567981 15937 leader_election.cc:288] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f35ad04a4ff4b2bb1ad4fcc524ac13a (127.27.229.194:37637), e8a92338991d4b6f8eef6cc6cd54b1b1 (127.27.229.195:40319)
I20260928 02:57:20.575161 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.568576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:60993 (local address 127.27.229.194:37637)
0928 02:57:20.568808 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.568813 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.569269 (+ 456us) server_negotiation.cc:407] Connection header received
0928 02:57:20.569591 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.569596 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.569675 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.569741 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:20.570514 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.571405 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.573294 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.574148 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.574346 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.574427 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.574717 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.574882 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:57:20.575524 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.569673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:32997 (local address 127.27.229.195:40319)
0928 02:57:20.569875 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.569881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.570025 (+ 144us) server_negotiation.cc:407] Connection header received
0928 02:57:20.570258 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.570263 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.570361 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.570468 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:20.571116 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.571921 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.573719 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.574495 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.574686 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.574764 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.575116 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.575250 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 02:57:20.575541 15939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.568442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:37637 (local address 127.27.229.193:60993)
0928 02:57:20.569125 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.569147 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.569303 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.569945 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.569948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.569976 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:20.570358 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.570367 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.571546 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.571551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.573137 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.573150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.574373 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.574379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.574538 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.574562 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.574690 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.574867 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"spinlock_wait_cycles":18688,"thread_start_us":387,"threads_started":1}
I20260928 02:57:20.575662 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.569286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:40319 (local address 127.27.229.193:32997)
0928 02:57:20.569917 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:20.569940 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:20.570119 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:20.570485 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:20.570489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:20.570513 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:20.570965 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.570976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.572056 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.572065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.573563 (+ 1498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:20.573574 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.574721 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.574727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:20.574942 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.574966 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:57:20.575076 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 02:57:20.575208 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":278,"threads_started":1}
I20260928 02:57:20.576792 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dca14d4734424f8e7a26a49498a13c" candidate_uuid: "49401898d8fa4a428392a57cf1c997f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" is_pre_election: true
I20260928 02:57:20.577291 15866 raft_consensus.cc:2454] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49401898d8fa4a428392a57cf1c997f2 in term 0.
I20260928 02:57:20.577993 15531 leader_election.cc:302] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49401898d8fa4a428392a57cf1c997f2, e8a92338991d4b6f8eef6cc6cd54b1b1; no voters:
I20260928 02:57:20.578006 15732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dca14d4734424f8e7a26a49498a13c" candidate_uuid: "49401898d8fa4a428392a57cf1c997f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" is_pre_election: true
I20260928 02:57:20.578444 15937 raft_consensus.cc:2781] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:20.578596 15937 raft_consensus.cc:482] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:20.578509 15732 raft_consensus.cc:2454] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 49401898d8fa4a428392a57cf1c997f2 in term 0.
I20260928 02:57:20.578696 15937 raft_consensus.cc:3037] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:20.581180 15937 raft_consensus.cc:504] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.581867 15937 leader_election.cc:288] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [CANDIDATE]: Term 1 election: Requested vote from peers 4f35ad04a4ff4b2bb1ad4fcc524ac13a (127.27.229.194:37637), e8a92338991d4b6f8eef6cc6cd54b1b1 (127.27.229.195:40319)
I20260928 02:57:20.582508 15732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dca14d4734424f8e7a26a49498a13c" candidate_uuid: "49401898d8fa4a428392a57cf1c997f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a"
I20260928 02:57:20.582463 15866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25dca14d4734424f8e7a26a49498a13c" candidate_uuid: "49401898d8fa4a428392a57cf1c997f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1"
I20260928 02:57:20.582739 15732 raft_consensus.cc:3037] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:20.582768 15866 raft_consensus.cc:3037] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:20.585150 15732 raft_consensus.cc:2454] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49401898d8fa4a428392a57cf1c997f2 in term 1.
I20260928 02:57:20.585150 15866 raft_consensus.cc:2454] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49401898d8fa4a428392a57cf1c997f2 in term 1.
I20260928 02:57:20.585624 15531 leader_election.cc:302] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49401898d8fa4a428392a57cf1c997f2, 4f35ad04a4ff4b2bb1ad4fcc524ac13a; no voters:
I20260928 02:57:20.585959 15937 raft_consensus.cc:2781] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:20.586431 15937 raft_consensus.cc:686] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 1 LEADER]: Becoming Leader. State: Replica: 49401898d8fa4a428392a57cf1c997f2, State: Running, Role: LEADER
I20260928 02:57:20.586989 15937 consensus_queue.cc:237] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } }
I20260928 02:57:20.591403 15452 catalog_manager.cc:5495] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 49401898d8fa4a428392a57cf1c997f2 (127.27.229.193). New cstate: current_term: 1 leader_uuid: "49401898d8fa4a428392a57cf1c997f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "49401898d8fa4a428392a57cf1c997f2" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38637 } health_report { overall_health: HEALTHY } } }
I20260928 02:57:20.667229 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.661568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.27.229.193:38637)
0928 02:57:20.661798 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.661806 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.661905 (+ 99us) server_negotiation.cc:407] Connection header received
0928 02:57:20.662171 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.662175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.662258 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.662417 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:20.662792 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.663590 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.665001 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.665404 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.665548 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.666081 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:20.666619 (+ 538us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:20.666786 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.666840 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.666952 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:57:20.675730 15866 raft_consensus.cc:1261] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 1 FOLLOWER]: Refusing update from remote peer 49401898d8fa4a428392a57cf1c997f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:20.675789 15732 raft_consensus.cc:1261] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 1 FOLLOWER]: Refusing update from remote peer 49401898d8fa4a428392a57cf1c997f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:20.676770 15941 consensus_queue.cc:1035] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f35ad04a4ff4b2bb1ad4fcc524ac13a" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 37637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:20.677165 15937 consensus_queue.cc:1035] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8a92338991d4b6f8eef6cc6cd54b1b1" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 40319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:20.704807 15453 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:20.705113 15453 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:20.709314 15453 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 25896212f3e44e95a37468d70759793c: Sending DeleteTablet for 3 replicas of tablet 25dca14d4734424f8e7a26a49498a13c
I20260928 02:57:20.711006 15846 tablet_service.cc:1514] Processing DeleteTablet for tablet 25dca14d4734424f8e7a26a49498a13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:35598
I20260928 02:57:20.710963 15578 tablet_service.cc:1514] Processing DeleteTablet for tablet 25dca14d4734424f8e7a26a49498a13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:45310
I20260928 02:57:20.710968 15712 tablet_service.cc:1514] Processing DeleteTablet for tablet 25dca14d4734424f8e7a26a49498a13c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:20 UTC) from {username='kudu', principal='kudu/127.27.229.254@KRBTEST.COM'} at 127.0.0.1:36288
I20260928 02:57:20.711984 15954 tablet_replica.cc:330] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: stopping tablet replica
I20260928 02:57:20.712477 15954 raft_consensus.cc:2229] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:20.713038 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15515
I20260928 02:57:20.713092 15954 raft_consensus.cc:2258] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:20.715787 15954 ts_tablet_manager.cc:1905] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:20.723582 15954 ts_tablet_manager.cc:1918] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:20.723881 15954 log.cc:1196] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/25dca14d4734424f8e7a26a49498a13c
I20260928 02:57:20.724565 15954 ts_tablet_manager.cc:1939] T 25dca14d4734424f8e7a26a49498a13c P e8a92338991d4b6f8eef6cc6cd54b1b1: Deleting consensus metadata
I20260928 02:57:20.726722 15438 catalog_manager.cc:4841] TS e8a92338991d4b6f8eef6cc6cd54b1b1 (127.27.229.195:40319): tablet 25dca14d4734424f8e7a26a49498a13c (table test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]) successfully deleted
I20260928 02:57:20.728417 15956 tablet_replica.cc:330] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: stopping tablet replica
I20260928 02:57:20.729542 15956 raft_consensus.cc:2229] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:20.730198 15956 raft_consensus.cc:2258] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:20.730863 15955 tablet_replica.cc:330] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: stopping tablet replica
I20260928 02:57:20.731631 15955 raft_consensus.cc:2229] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:20.732400 15955 raft_consensus.cc:2258] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:20.733301 15956 ts_tablet_manager.cc:1905] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:20.734520 15955 ts_tablet_manager.cc:1905] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 28 02:57:20 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.193@KRBTEST.COM
I20260928 02:57:20.741245 15956 ts_tablet_manager.cc:1918] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:20.741427 15956 log.cc:1196] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/25dca14d4734424f8e7a26a49498a13c
I20260928 02:57:20.742040 15956 ts_tablet_manager.cc:1939] T 25dca14d4734424f8e7a26a49498a13c P 4f35ad04a4ff4b2bb1ad4fcc524ac13a: Deleting consensus metadata
I20260928 02:57:20.743253 15955 ts_tablet_manager.cc:1918] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:20.743505 15955 log.cc:1196] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/25dca14d4734424f8e7a26a49498a13c
I20260928 02:57:20.744191 15955 ts_tablet_manager.cc:1939] T 25dca14d4734424f8e7a26a49498a13c P 49401898d8fa4a428392a57cf1c997f2: Deleting consensus metadata
I20260928 02:57:20.744179 15438 catalog_manager.cc:4841] TS 4f35ad04a4ff4b2bb1ad4fcc524ac13a (127.27.229.194:37637): tablet 25dca14d4734424f8e7a26a49498a13c (table test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]) successfully deleted
I20260928 02:57:20.746342 15438 catalog_manager.cc:4841] TS 49401898d8fa4a428392a57cf1c997f2 (127.27.229.193:38637): tablet 25dca14d4734424f8e7a26a49498a13c (table test-table [id=fbf94fa77b3b41fa8723d7832a5887f8]) successfully deleted
I20260928 02:57:20.749159 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.727144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.27.229.193:38637)
0928 02:57:20.727883 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.727893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.727935 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:57:20.728027 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.728036 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.728172 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.728323 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:20.731184 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.732578 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.734733 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.735646 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.735759 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.741720 (+ 5961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:20.741740 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:20.741764 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:20.741842 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:20.745760 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.746783 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.746792 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.746795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.747027 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.747450 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.747455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.747460 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.747824 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:20.748173 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.748511 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.748778 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553}
I20260928 02:57:20.749902 15638 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:20.851718 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15515
I20260928 02:57:20.872419 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15649
Sep 28 02:57:20 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.194@KRBTEST.COM
I20260928 02:57:20.890372 15925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:20.873262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.27.229.194:37637)
0928 02:57:20.873475 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:20.873488 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:20.873537 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:57:20.873764 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:20.873768 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:20.873843 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:20.873932 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:20.875325 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.876199 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.878006 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:20.878406 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:20.878526 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:20.883682 (+ 5156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:20.883700 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:20.883718 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:20.883765 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:20.887141 (+ 3376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.887981 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.887989 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.887992 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.888133 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:20.888556 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:20.888561 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:20.888564 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:20.889009 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:20.889361 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:20.889725 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:57:20.889977 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 02:57:20.891162 15772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:20.981980 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15649
I20260928 02:57:20.999181 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15783
Sep 28 02:57:21 dist-test-slave-s1kg krb5kdc[15409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.195@KRBTEST.COM
I20260928 02:57:21.015534 15927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:21.000062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.27.229.195:40319)
0928 02:57:21.000240 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:21.000245 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:21.000263 (+ 18us) server_negotiation.cc:407] Connection header received
0928 02:57:21.000416 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:21.000469 (+ 53us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:21.000596 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:21.000708 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:21.002052 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:21.002805 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:21.004172 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:21.004603 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:21.004701 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:21.009280 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:21.009296 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:21.009312 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:21.009351 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:21.012386 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:21.013135 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:21.013143 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:21.013145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:21.013300 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:21.013827 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:21.013831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:21.013834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:21.014256 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:21.014518 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:21.014814 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 02:57:21.015148 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:57:21.016229 15906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:21.106813 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15783
I20260928 02:57:21.123474 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15422
I20260928 02:57:21.124629 15484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:21.237142 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15422
2026-09-28T02:57:21Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T02:57:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:21Z Disabled control of system clock
I20260928 02:57:21.318351 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39857
--webserver_interface=127.27.229.254
--webserver_port=0
--builtin_ntp_servers=127.27.229.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:21.678210 15965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:21.678536 15965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:21.678622 15965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:21.686029 15965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:21.686151 15965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:21.686188 15965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:21.686239 15965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:21.686283 15965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:21.695701 15965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39857
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:21.696964 15965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:21.698786 15965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:21.710798 15973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:21.710819 15971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:21.710834 15970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:21.711719 15965 server_base.cc:1034] running on GCE node
I20260928 02:57:21.712489 15965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:21.713860 15965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:21.715054 15965 hybrid_clock.cc:648] HybridClock initialized: now 1790564241715016 us; error 28 us; skew 500 ppm
I20260928 02:57:21.715365 15965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:21.721000 15965 webserver.cc:460] Webserver started at http://127.27.229.254:35297/ using document root <none> and password file <none>
I20260928 02:57:21.721691 15965 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:21.721781 15965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:21.722061 15965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:21.725210 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "8608701d85fd4eb5850728953ce91486"
format_stamp: "Formatted at 2026-09-28 02:57:21 on dist-test-slave-s1kg"
server_key: "c346373bccfcbf1b5e362999d7910879"
server_key_iv: "20f63fae6f1747eafde281f36706a8a3"
server_key_version: "encryptionkey@0"
I20260928 02:57:21.726002 15965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "8608701d85fd4eb5850728953ce91486"
format_stamp: "Formatted at 2026-09-28 02:57:21 on dist-test-slave-s1kg"
server_key: "c346373bccfcbf1b5e362999d7910879"
server_key_iv: "20f63fae6f1747eafde281f36706a8a3"
server_key_version: "encryptionkey@0"
I20260928 02:57:21.731590 15965 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260928 02:57:21.735219 15979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:21.736661 15965 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:57:21.736825 15965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "8608701d85fd4eb5850728953ce91486"
format_stamp: "Formatted at 2026-09-28 02:57:21 on dist-test-slave-s1kg"
server_key: "c346373bccfcbf1b5e362999d7910879"
server_key_iv: "20f63fae6f1747eafde281f36706a8a3"
server_key_version: "encryptionkey@0"
I20260928 02:57:21.737017 15965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:21.754065 15965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:21.755048 15965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:21.755297 15965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:21.776403 15965 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:39857
I20260928 02:57:21.776417 16030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:39857 every 8 connection(s)
I20260928 02:57:21.777915 15965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:21.781822 16031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:21.785249 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 15965
I20260928 02:57:21.785470 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:21.785910 28567 external_mini_cluster.cc:1428] Setting key e96c1d11e6d69531741c03b3fdbb2253
I20260928 02:57:21.794296 16031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: Bootstrap starting.
I20260928 02:57:21.797793 16031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:21.799077 16031 log.cc:824] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:21.799996 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:21.787872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42142 (local address 127.27.229.254:39857)
0928 02:57:21.788876 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:21.788889 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:21.788932 (+ 43us) server_negotiation.cc:407] Connection header received
0928 02:57:21.790688 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:21.790711 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:21.791319 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:21.791770 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:21.792389 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:21.793710 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:21.795140 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:21.795857 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:21.796069 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:21.797268 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:21.797276 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:21.797291 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:21.797860 (+ 569us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:21.797991 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:21.798020 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:21.798151 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:21.798369 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 02:57:21.798685 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":367,"threads_started":1}
I20260928 02:57:21.802397 16031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: No bootstrap required, opened a new log
I20260928 02:57:21.810554 16031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } }
I20260928 02:57:21.811043 16031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:21.811170 16031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8608701d85fd4eb5850728953ce91486, State: Initialized, Role: FOLLOWER
I20260928 02:57:21.811851 16031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } }
I20260928 02:57:21.812129 16031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:21.812278 16031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:21.812453 16031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:21.814631 16031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } }
I20260928 02:57:21.815205 16031 leader_election.cc:302] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8608701d85fd4eb5850728953ce91486; no voters:
I20260928 02:57:21.815807 16031 leader_election.cc:288] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:21.815980 16036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:21.816731 16036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [term 1 LEADER]: Becoming Leader. State: Replica: 8608701d85fd4eb5850728953ce91486, State: Running, Role: LEADER
I20260928 02:57:21.817648 16036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } }
I20260928 02:57:21.818200 16031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:21.820428 16037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8608701d85fd4eb5850728953ce91486. Latest consensus state: current_term: 1 leader_uuid: "8608701d85fd4eb5850728953ce91486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } } }
I20260928 02:57:21.820511 16038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8608701d85fd4eb5850728953ce91486" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8608701d85fd4eb5850728953ce91486" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39857 } } }
I20260928 02:57:21.820843 16037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:21.820904 16038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:21.822654 16042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:21.829109 16042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:21.838611 16042 catalog_manager.cc:1347] Generated new cluster ID: ebf2fcc76ec3490585afd19ff815a65a
I20260928 02:57:21.838754 16042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:21.860138 16042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:21.862259 16042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:21.875190 16042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: Generated new TSK 0
I20260928 02:57:21.876451 16042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:21.884991 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.193:0
--local_ip_for_outbound_sockets=127.27.229.193
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--builtin_ntp_servers=127.27.229.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 02:57:22.237450 16055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:22.237753 16055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:22.237897 16055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:22.246155 16055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:22.246268 16055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:22.246438 16055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.193
I20260928 02:57:22.256002 16055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.229.193
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.193
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:22.257334 16055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:22.259256 16055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:22.273022 16061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:22.273144 16063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:22.273020 16060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:22.273739 16055 server_base.cc:1034] running on GCE node
I20260928 02:57:22.274431 16055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:22.275542 16055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:22.276798 16055 hybrid_clock.cc:648] HybridClock initialized: now 1790564242276727 us; error 61 us; skew 500 ppm
I20260928 02:57:22.277148 16055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:22.279824 16055 webserver.cc:460] Webserver started at http://127.27.229.193:34265/ using document root <none> and password file <none>
I20260928 02:57:22.280565 16055 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:22.280700 16055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:22.281029 16055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:22.284144 16055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data/instance:
uuid: "23b1728464204cb48c7fa68f924d02cb"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "c21e4c7948648fd2d9708bd379ab7710"
server_key_iv: "b9311894fbcbb1542d0fda976803cf76"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.285025 16055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance:
uuid: "23b1728464204cb48c7fa68f924d02cb"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "c21e4c7948648fd2d9708bd379ab7710"
server_key_iv: "b9311894fbcbb1542d0fda976803cf76"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.290514 16055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:57:22.294435 16069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.295935 16055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:22.296137 16055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal
uuid: "23b1728464204cb48c7fa68f924d02cb"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "c21e4c7948648fd2d9708bd379ab7710"
server_key_iv: "b9311894fbcbb1542d0fda976803cf76"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.296379 16055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:22.311836 16055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:22.312753 16055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:22.313084 16055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:22.314093 16055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:22.315681 16055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:22.315804 16055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.315941 16055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:22.316028 16055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.350749 16055 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.193:38993
I20260928 02:57:22.350803 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.193:38993 every 8 connection(s)
I20260928 02:57:22.352114 16055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/data/info.pb
I20260928 02:57:22.354825 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16055
I20260928 02:57:22.355047 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal/instance
I20260928 02:57:22.355607 28567 external_mini_cluster.cc:1428] Setting key e8346653624ea5f8f35aa1f953815d3a
I20260928 02:57:22.358812 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.194:0
--local_ip_for_outbound_sockets=127.27.229.194
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--builtin_ntp_servers=127.27.229.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:22.367882 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:22.354878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.193:57093 (local address 127.27.229.254:39857)
0928 02:57:22.355373 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:22.355379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:22.356829 (+ 1450us) server_negotiation.cc:407] Connection header received
0928 02:57:22.359397 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:22.359402 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:22.359481 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:22.359572 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:22.361195 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.361924 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.364575 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.365034 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.365148 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:22.367043 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:22.367048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:22.367051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:22.367071 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:22.367118 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:22.367131 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:22.367213 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:22.367513 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 02:57:22.367610 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":240,"threads_started":1}
I20260928 02:57:22.369071 16185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:22.355516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39857 (local address 127.27.229.193:57093)
0928 02:57:22.356657 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:22.356706 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:22.359032 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:22.359885 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:22.359901 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:22.360054 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:22.360941 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:22.360984 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.362101 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.362109 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:22.364375 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:22.364397 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.365293 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.365302 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:22.365656 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:22.366553 (+ 897us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:22.366560 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:22.366621 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:22.366757 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:22.367239 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:22.367254 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:22.367265 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 02:57:22.367461 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:57:22.367986 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":413,"threads_started":1}
I20260928 02:57:22.370646 16182 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39857
I20260928 02:57:22.371023 16182 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:22.372023 16182 heartbeater.cc:511] Master 127.27.229.254:39857 requested a full tablet report, sending...
I20260928 02:57:22.374568 15996 ts_manager.cc:194] Registered new tserver with Master: 23b1728464204cb48c7fa68f924d02cb (127.27.229.193:38993)
I20260928 02:57:22.376286 15996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.193:57093
W20260928 02:57:22.700356 16186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:22.700647 16186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:22.700784 16186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:22.708287 16186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:22.708403 16186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:22.708585 16186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.194
I20260928 02:57:22.717975 16186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.229.194
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.194
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:22.719419 16186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:22.721401 16186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:22.736630 16191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:22.736570 16194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:22.736613 16192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:22.738144 16186 server_base.cc:1034] running on GCE node
I20260928 02:57:22.739028 16186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:22.740310 16186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:22.741582 16186 hybrid_clock.cc:648] HybridClock initialized: now 1790564242741525 us; error 51 us; skew 500 ppm
I20260928 02:57:22.742094 16186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:22.744678 16186 webserver.cc:460] Webserver started at http://127.27.229.194:45693/ using document root <none> and password file <none>
I20260928 02:57:22.745433 16186 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:22.745568 16186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:22.745886 16186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:22.748996 16186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data/instance:
uuid: "699dbd1a1f6546918cd3278c61d6b16c"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "8b0a0a12ab32b30505eb1ef2b315ae8a"
server_key_iv: "c8c00390ad0ec8043cf62fa068c2b60a"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.749802 16186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance:
uuid: "699dbd1a1f6546918cd3278c61d6b16c"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "8b0a0a12ab32b30505eb1ef2b315ae8a"
server_key_iv: "c8c00390ad0ec8043cf62fa068c2b60a"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.755296 16186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260928 02:57:22.759212 16200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.760669 16186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:22.760851 16186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal
uuid: "699dbd1a1f6546918cd3278c61d6b16c"
format_stamp: "Formatted at 2026-09-28 02:57:22 on dist-test-slave-s1kg"
server_key: "8b0a0a12ab32b30505eb1ef2b315ae8a"
server_key_iv: "c8c00390ad0ec8043cf62fa068c2b60a"
server_key_version: "encryptionkey@0"
I20260928 02:57:22.761083 16186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:22.785673 16186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:22.786543 16186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:22.786890 16186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:22.787914 16186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:22.789461 16186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:22.789584 16186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.789701 16186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:22.789788 16186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:22.824075 16186 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.194:45505
I20260928 02:57:22.824165 16312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.194:45505 every 8 connection(s)
I20260928 02:57:22.825516 16186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/data/info.pb
I20260928 02:57:22.829751 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16186
I20260928 02:57:22.829941 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal/instance
I20260928 02:57:22.830523 28567 external_mini_cluster.cc:1428] Setting key a12020388118992f2fc134d8993f84a0
I20260928 02:57:22.833760 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.229.195:0
--local_ip_for_outbound_sockets=127.27.229.195
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--builtin_ntp_servers=127.27.229.212:44349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 02:57:22.840700 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:22.828356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:47251 (local address 127.27.229.254:39857)
0928 02:57:22.828549 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:22.828553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:22.830181 (+ 1628us) server_negotiation.cc:407] Connection header received
0928 02:57:22.832477 (+ 2296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:22.832482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:22.832563 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:22.832671 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:22.834200 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.835015 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.837515 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.837916 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.838027 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:22.839715 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:22.839720 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:22.839723 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:22.839743 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:22.839776 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:22.839787 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:22.839860 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:22.840190 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 02:57:22.840339 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:57:22.841980 16315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:22.828833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39857 (local address 127.27.229.194:47251)
0928 02:57:22.829977 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:22.830056 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:22.832168 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:22.832990 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:22.833002 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:22.833135 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:22.834033 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:22.834061 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.835173 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.835182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:22.837348 (+ 2166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:22.837365 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:22.838166 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:22.838175 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:22.838504 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:22.839330 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:22.839336 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:22.839400 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:22.839518 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:22.839886 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:22.839899 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:22.839911 (+ 12us) client_negotiation.cc:769] Sending connection context
0928 02:57:22.840123 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 02:57:22.840561 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":288,"threads_started":1}
I20260928 02:57:22.843591 16313 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39857
I20260928 02:57:22.843910 16313 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:22.844676 16313 heartbeater.cc:511] Master 127.27.229.254:39857 requested a full tablet report, sending...
I20260928 02:57:22.846168 15996 ts_manager.cc:194] Registered new tserver with Master: 699dbd1a1f6546918cd3278c61d6b16c (127.27.229.194:45505)
I20260928 02:57:22.847062 15996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.194:47251
W20260928 02:57:23.177120 16316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:23.177558 16316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:23.177712 16316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:23.185369 16316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:23.185537 16316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:23.185707 16316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.229.195
I20260928 02:57:23.195621 16316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:44349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.229.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.229.195
--webserver_port=0
--tserver_master_addrs=127.27.229.254:39857
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.229.195
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:23.196991 16316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:23.198793 16316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:23.212688 16321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:23.212798 16322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:23.212772 16324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:23.213814 16316 server_base.cc:1034] running on GCE node
I20260928 02:57:23.214490 16316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:23.215637 16316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:23.216883 16316 hybrid_clock.cc:648] HybridClock initialized: now 1790564243216826 us; error 49 us; skew 500 ppm
I20260928 02:57:23.217232 16316 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260928 02:57:23.219951 16316 webserver.cc:460] Webserver started at http://127.27.229.195:41783/ using document root <none> and password file <none>
I20260928 02:57:23.220693 16316 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:23.220824 16316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:23.221223 16316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:23.224737 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data/instance:
uuid: "feb4b6069fa44de4adb52d343b9ce101"
format_stamp: "Formatted at 2026-09-28 02:57:23 on dist-test-slave-s1kg"
server_key: "b46f023df9289b4d27cb0671b5bdb30b"
server_key_iv: "af020c34ff74a2212c1b066d6e069447"
server_key_version: "encryptionkey@0"
I20260928 02:57:23.225732 16316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance:
uuid: "feb4b6069fa44de4adb52d343b9ce101"
format_stamp: "Formatted at 2026-09-28 02:57:23 on dist-test-slave-s1kg"
server_key: "b46f023df9289b4d27cb0671b5bdb30b"
server_key_iv: "af020c34ff74a2212c1b066d6e069447"
server_key_version: "encryptionkey@0"
I20260928 02:57:23.231334 16316 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:57:23.235211 16330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:23.236711 16316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:23.236907 16316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal
uuid: "feb4b6069fa44de4adb52d343b9ce101"
format_stamp: "Formatted at 2026-09-28 02:57:23 on dist-test-slave-s1kg"
server_key: "b46f023df9289b4d27cb0671b5bdb30b"
server_key_iv: "af020c34ff74a2212c1b066d6e069447"
server_key_version: "encryptionkey@0"
I20260928 02:57:23.237155 16316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:23.252631 16316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:23.253556 16316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:23.253885 16316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:23.255036 16316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 02:57:23.256623 16316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 02:57:23.256759 16316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:23.256893 16316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 02:57:23.256980 16316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:23.293264 16316 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.195:43023
I20260928 02:57:23.293354 16442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.195:43023 every 8 connection(s)
I20260928 02:57:23.294633 16316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/data/info.pb
I20260928 02:57:23.303139 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16316
I20260928 02:57:23.303323 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal/instance
I20260928 02:57:23.304055 28567 external_mini_cluster.cc:1428] Setting key 9e452817d302b1670de12c5b9f979921
I20260928 02:57:23.308703 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.297520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.195:40543 (local address 127.27.229.254:39857)
0928 02:57:23.297762 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.297769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.299189 (+ 1420us) server_negotiation.cc:407] Connection header received
0928 02:57:23.301073 (+ 1884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.301079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.301189 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.301315 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:23.302453 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.303675 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.305708 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.306206 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.306349 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.307818 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:23.307824 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:23.307827 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:23.307846 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:23.307877 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:23.307888 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:23.307992 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.308244 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.308371 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 02:57:23.309773 16445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.298037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39857 (local address 127.27.229.195:40543)
0928 02:57:23.299027 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.299064 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.300724 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.301558 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.301567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.301659 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:57:23.302250 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.302268 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.303953 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.303957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.305538 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.305552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.306442 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.306450 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.306714 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.307480 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 02:57:23.307483 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:57:23.307525 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 02:57:23.307614 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:57:23.307983 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:57:23.307994 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:57:23.308004 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.308223 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.308692 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":280,"threads_started":1}
I20260928 02:57:23.311287 16443 heartbeater.cc:347] Connected to a master server at 127.27.229.254:39857
I20260928 02:57:23.311650 16443 heartbeater.cc:464] Registering TS with master...
I20260928 02:57:23.312465 16443 heartbeater.cc:511] Master 127.27.229.254:39857 requested a full tablet report, sending...
I20260928 02:57:23.314159 15996 ts_manager.cc:194] Registered new tserver with Master: feb4b6069fa44de4adb52d343b9ce101 (127.27.229.195:43023)
I20260928 02:57:23.315068 15996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.229.195:40543
I20260928 02:57:23.320263 28567 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 02:57:23.331032 16184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.324727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.27.229.254:39857)
0928 02:57:23.325085 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.325091 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.325516 (+ 425us) server_negotiation.cc:407] Connection header received
0928 02:57:23.325593 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.325597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.325693 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.325854 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:23.326301 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.327106 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.328565 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.328989 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.329102 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.330186 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:23.330192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:23.330195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:23.330217 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:23.330284 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:23.330307 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:23.330420 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.330586 (+ 166us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.330691 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260928 02:57:23.335422 15996 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42144:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 02:57:23.343401 15996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 02:57:23.370663 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.356994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56402)
0928 02:57:23.358971 (+ 1977us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.358999 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.359231 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.360319 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.360330 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.360363 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:23.360806 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.360830 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.363037 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.363042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.366972 (+ 3930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.366996 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.369578 (+ 2582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.369586 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.369805 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.369862 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.370014 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.370237 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1784,"mutex_wait_us":121,"thread_start_us":1611,"threads_started":1}
I20260928 02:57:23.371232 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.356535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43023 (local address 127.0.0.1:43478)
0928 02:57:23.357642 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.357696 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.357911 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.360946 (+ 3035us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.360951 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.360982 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:23.361368 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.361381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.364874 (+ 3493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.364884 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.367301 (+ 2417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.367315 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.370345 (+ 3030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.370352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.370546 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.370589 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.370676 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.370853 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":1,"thread_start_us":656,"threads_started":1}
I20260928 02:57:23.371800 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.357466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56402 (local address 127.27.229.193:38993)
0928 02:57:23.358600 (+ 1134us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.358622 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.359119 (+ 497us) server_negotiation.cc:407] Connection header received
0928 02:57:23.359452 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.359464 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.359720 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.359912 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:23.361048 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.362790 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.367220 (+ 4430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.368941 (+ 1721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.369136 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.369382 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.371202 (+ 1820us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.371409 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":931,"thread_start_us":811,"threads_started":1}
I20260928 02:57:23.372470 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.356834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43478 (local address 127.27.229.195:43023)
0928 02:57:23.360306 (+ 3472us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.360315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.360348 (+ 33us) server_negotiation.cc:407] Connection header received
0928 02:57:23.360444 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.360455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.360711 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.361580 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:23.361855 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.364702 (+ 2847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.368632 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.370126 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.371544 (+ 1418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.371766 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.371845 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.372024 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2138,"thread_start_us":1994,"threads_started":1}
I20260928 02:57:23.376570 16117 tablet_service.cc:1467] Processing CreateTablet for tablet c3542766707f4fb7b8d218df448b03d4 (DEFAULT_TABLE table=test-table [id=ec08d0b10ac846ab923b103cbabc041e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:23.376570 16378 tablet_service.cc:1467] Processing CreateTablet for tablet c3542766707f4fb7b8d218df448b03d4 (DEFAULT_TABLE table=test-table [id=ec08d0b10ac846ab923b103cbabc041e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:23.378248 16378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3542766707f4fb7b8d218df448b03d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:23.378249 16117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3542766707f4fb7b8d218df448b03d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:23.380339 16182 heartbeater.cc:503] Master 127.27.229.254:39857 was elected leader, sending a full tablet report...
I20260928 02:57:23.382696 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.359038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.194:45505 (local address 127.0.0.1:52904)
0928 02:57:23.364902 (+ 5864us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.364926 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.365081 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.375289 (+ 10208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.375294 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.375324 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:23.375670 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.375686 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.377071 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.377076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.378534 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.378548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.381696 (+ 3148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.381705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.382063 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.382089 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.382198 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.382352 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5708,"thread_start_us":5573,"threads_started":1}
I20260928 02:57:23.382735 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.359493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.27.229.194:45505)
0928 02:57:23.374718 (+ 15225us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.374724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.374750 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:57:23.374889 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.374897 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.375104 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.375299 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:23.375892 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.376932 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.378967 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.381162 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.381341 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.381550 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.382247 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.382428 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15090,"thread_start_us":15989,"threads_started":1}
I20260928 02:57:23.386319 16248 tablet_service.cc:1467] Processing CreateTablet for tablet c3542766707f4fb7b8d218df448b03d4 (DEFAULT_TABLE table=test-table [id=ec08d0b10ac846ab923b103cbabc041e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 02:57:23.387883 16248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3542766707f4fb7b8d218df448b03d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:23.396955 16462 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Bootstrap starting.
I20260928 02:57:23.397369 16461 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Bootstrap starting.
I20260928 02:57:23.400265 16463 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Bootstrap starting.
I20260928 02:57:23.400859 16461 tablet_bootstrap.cc:654] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:23.400905 16462 tablet_bootstrap.cc:654] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:23.402064 16462 log.cc:824] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:23.402312 16461 log.cc:824] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:23.404022 16463 tablet_bootstrap.cc:654] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:23.405119 16461 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: No bootstrap required, opened a new log
I20260928 02:57:23.405395 16461 ts_tablet_manager.cc:1397] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260928 02:57:23.405357 16463 log.cc:824] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:23.405498 16462 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: No bootstrap required, opened a new log
I20260928 02:57:23.405781 16462 ts_tablet_manager.cc:1397] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 02:57:23.410812 16463 tablet_bootstrap.cc:492] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: No bootstrap required, opened a new log
I20260928 02:57:23.411160 16463 ts_tablet_manager.cc:1397] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260928 02:57:23.413540 16461 raft_consensus.cc:348] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.413926 16461 raft_consensus.cc:374] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:23.414028 16461 raft_consensus.cc:729] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b1728464204cb48c7fa68f924d02cb, State: Initialized, Role: FOLLOWER
I20260928 02:57:23.414636 16461 consensus_queue.cc:260] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.416448 16461 ts_tablet_manager.cc:1428] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260928 02:57:23.417608 16462 raft_consensus.cc:348] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.418135 16462 raft_consensus.cc:374] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:23.418247 16462 raft_consensus.cc:729] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb4b6069fa44de4adb52d343b9ce101, State: Initialized, Role: FOLLOWER
I20260928 02:57:23.418946 16462 consensus_queue.cc:260] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.420114 16443 heartbeater.cc:503] Master 127.27.229.254:39857 was elected leader, sending a full tablet report...
I20260928 02:57:23.420751 16462 ts_tablet_manager.cc:1428] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260928 02:57:23.422878 16463 raft_consensus.cc:348] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.423411 16463 raft_consensus.cc:374] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:23.423520 16463 raft_consensus.cc:729] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 699dbd1a1f6546918cd3278c61d6b16c, State: Initialized, Role: FOLLOWER
I20260928 02:57:23.424131 16463 consensus_queue.cc:260] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.425319 16313 heartbeater.cc:503] Master 127.27.229.254:39857 was elected leader, sending a full tablet report...
I20260928 02:57:23.425772 16463 ts_tablet_manager.cc:1428] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260928 02:57:23.426326 16469 raft_consensus.cc:482] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 02:57:23.426668 16469 raft_consensus.cc:504] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.428406 16469 leader_election.cc:288] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb4b6069fa44de4adb52d343b9ce101 (127.27.229.195:43023), 23b1728464204cb48c7fa68f924d02cb (127.27.229.193:38993)
I20260928 02:57:23.435454 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.428744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:38175 (local address 127.27.229.195:43023)
0928 02:57:23.428923 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.428928 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.429511 (+ 583us) server_negotiation.cc:407] Connection header received
0928 02:57:23.429840 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.429846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.429927 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.430027 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:23.430916 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.431861 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.433718 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.434516 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.434684 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.434755 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.435092 (+ 337us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.435215 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 02:57:23.435563 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.428772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.195:43023 (local address 127.27.229.194:38175)
0928 02:57:23.429413 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.429431 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.429622 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.430198 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.430204 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.430233 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:23.430719 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.430737 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.431985 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.431992 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.433537 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.433550 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.434709 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.434715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.434923 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.434955 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.435069 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.435205 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"mutex_wait_us":24,"thread_start_us":701,"threads_started":1}
I20260928 02:57:23.435925 16471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.429252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.27.229.194:58927)
0928 02:57:23.430027 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.430043 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 02:57:23.430156 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:57:23.430515 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:57:23.430519 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:57:23.430542 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 02:57:23.430968 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.430983 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.432121 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.432131 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.433719 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:57:23.433732 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.435051 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.435059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:57:23.435233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.435257 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 02:57:23.435364 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 02:57:23.435503 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":30,"thread_start_us":458,"threads_started":1}
I20260928 02:57:23.436385 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.429409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.229.194:58927 (local address 127.27.229.193:38993)
0928 02:57:23.429680 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.429689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.430114 (+ 425us) server_negotiation.cc:407] Connection header received
0928 02:57:23.430323 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.430328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.430402 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.430501 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 02:57:23.431141 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.431962 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.433909 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.434774 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.435034 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.435107 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.435394 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.435549 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:57:23.436650 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3542766707f4fb7b8d218df448b03d4" candidate_uuid: "699dbd1a1f6546918cd3278c61d6b16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4b6069fa44de4adb52d343b9ce101" is_pre_election: true
I20260928 02:57:23.437300 16398 raft_consensus.cc:2454] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699dbd1a1f6546918cd3278c61d6b16c in term 0.
I20260928 02:57:23.437291 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3542766707f4fb7b8d218df448b03d4" candidate_uuid: "699dbd1a1f6546918cd3278c61d6b16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b1728464204cb48c7fa68f924d02cb" is_pre_election: true
I20260928 02:57:23.437865 16137 raft_consensus.cc:2454] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 699dbd1a1f6546918cd3278c61d6b16c in term 0.
I20260928 02:57:23.437985 16204 leader_election.cc:302] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 699dbd1a1f6546918cd3278c61d6b16c, feb4b6069fa44de4adb52d343b9ce101; no voters:
I20260928 02:57:23.438457 16469 raft_consensus.cc:2781] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 02:57:23.438632 16469 raft_consensus.cc:482] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 02:57:23.438748 16469 raft_consensus.cc:3037] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:23.441342 16469 raft_consensus.cc:504] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.442086 16469 leader_election.cc:288] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [CANDIDATE]: Term 1 election: Requested vote from peers feb4b6069fa44de4adb52d343b9ce101 (127.27.229.195:43023), 23b1728464204cb48c7fa68f924d02cb (127.27.229.193:38993)
I20260928 02:57:23.442513 16398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3542766707f4fb7b8d218df448b03d4" candidate_uuid: "699dbd1a1f6546918cd3278c61d6b16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb4b6069fa44de4adb52d343b9ce101"
I20260928 02:57:23.442649 16137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3542766707f4fb7b8d218df448b03d4" candidate_uuid: "699dbd1a1f6546918cd3278c61d6b16c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23b1728464204cb48c7fa68f924d02cb"
I20260928 02:57:23.442730 16398 raft_consensus.cc:3037] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:23.442919 16137 raft_consensus.cc:3037] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:23.445250 16137 raft_consensus.cc:2454] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699dbd1a1f6546918cd3278c61d6b16c in term 1.
I20260928 02:57:23.445252 16398 raft_consensus.cc:2454] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 699dbd1a1f6546918cd3278c61d6b16c in term 1.
I20260928 02:57:23.445688 16203 leader_election.cc:302] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23b1728464204cb48c7fa68f924d02cb, 699dbd1a1f6546918cd3278c61d6b16c; no voters:
I20260928 02:57:23.446053 16469 raft_consensus.cc:2781] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:23.446487 16469 raft_consensus.cc:686] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 1 LEADER]: Becoming Leader. State: Replica: 699dbd1a1f6546918cd3278c61d6b16c, State: Running, Role: LEADER
I20260928 02:57:23.447053 16469 consensus_queue.cc:237] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } }
I20260928 02:57:23.451745 15995 catalog_manager.cc:5495] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c reported cstate change: term changed from 0 to 1, leader changed from <none> to 699dbd1a1f6546918cd3278c61d6b16c (127.27.229.194). New cstate: current_term: 1 leader_uuid: "699dbd1a1f6546918cd3278c61d6b16c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "699dbd1a1f6546918cd3278c61d6b16c" member_type: VOTER last_known_addr { host: "127.27.229.194" port: 45505 } health_report { overall_health: HEALTHY } } }
I20260928 02:57:23.470033 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.464558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.27.229.194:45505)
0928 02:57:23.464758 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.464762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.464781 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:57:23.464977 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.464987 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.465089 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.465213 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:57:23.465673 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.466404 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.467838 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.468236 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.468339 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.468837 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:57:23.469351 (+ 514us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:57:23.469461 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.469640 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.469706 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:57:23.479082 16137 raft_consensus.cc:1261] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 1 FOLLOWER]: Refusing update from remote peer 699dbd1a1f6546918cd3278c61d6b16c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:23.479082 16398 raft_consensus.cc:1261] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 1 FOLLOWER]: Refusing update from remote peer 699dbd1a1f6546918cd3278c61d6b16c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:57:23.480379 16472 consensus_queue.cc:1035] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [LEADER]: Connected to new peer: Peer: permanent_uuid: "23b1728464204cb48c7fa68f924d02cb" member_type: VOTER last_known_addr { host: "127.27.229.193" port: 38993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:23.480811 16469 consensus_queue.cc:1035] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb4b6069fa44de4adb52d343b9ce101" member_type: VOTER last_known_addr { host: "127.27.229.195" port: 43023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:57:23.508072 15995 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:23.508365 15995 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42144:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 02:57:23.512141 15995 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8608701d85fd4eb5850728953ce91486: Sending DeleteTablet for 3 replicas of tablet c3542766707f4fb7b8d218df448b03d4
I20260928 02:57:23.513365 16378 tablet_service.cc:1514] Processing DeleteTablet for tablet c3542766707f4fb7b8d218df448b03d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:23 UTC) from {username='slave'} at 127.0.0.1:43478
I20260928 02:57:23.513543 16117 tablet_service.cc:1514] Processing DeleteTablet for tablet c3542766707f4fb7b8d218df448b03d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:23 UTC) from {username='slave'} at 127.0.0.1:56402
I20260928 02:57:23.514602 16484 tablet_replica.cc:330] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: stopping tablet replica
I20260928 02:57:23.514727 16248 tablet_service.cc:1514] Processing DeleteTablet for tablet c3542766707f4fb7b8d218df448b03d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 02:57:23 UTC) from {username='slave'} at 127.0.0.1:52904
I20260928 02:57:23.515106 16484 raft_consensus.cc:2229] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:23.515445 16485 tablet_replica.cc:330] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: stopping tablet replica
I20260928 02:57:23.515537 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16055
I20260928 02:57:23.515678 16484 raft_consensus.cc:2258] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:23.515987 16485 raft_consensus.cc:2229] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 02:57:23.516486 16485 raft_consensus.cc:2258] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:23.519604 16485 ts_tablet_manager.cc:1905] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:23.525460 16484 ts_tablet_manager.cc:1905] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:23.528137 16486 tablet_replica.cc:330] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: stopping tablet replica
I20260928 02:57:23.528177 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.516747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56412 (local address 127.27.229.193:38993)
0928 02:57:23.516942 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.516949 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.517570 (+ 621us) server_negotiation.cc:407] Connection header received
0928 02:57:23.517885 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.517913 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.518015 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.518097 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:23.518781 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.519994 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.522075 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.522786 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.522932 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.526261 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:23.526275 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:23.526293 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:23.527171 (+ 878us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:23.527242 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:23.527270 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:23.527374 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.527638 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.527767 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:57:23.528582 16486 raft_consensus.cc:2229] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 1 LEADER]: Raft consensus shutting down.
I20260928 02:57:23.529186 16485 ts_tablet_manager.cc:1918] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:23.529312 16486 raft_consensus.cc:2258] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 02:57:23.529371 16485 log.cc:1196] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-0/wal/wals/c3542766707f4fb7b8d218df448b03d4
I20260928 02:57:23.529692 16177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:23.536645 16486 ts_tablet_manager.cc:1905] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 02:57:23.537329 16484 ts_tablet_manager.cc:1918] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:23.537698 16484 log.cc:1196] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-2/wal/wals/c3542766707f4fb7b8d218df448b03d4
I20260928 02:57:23.538305 16484 ts_tablet_manager.cc:1939] T c3542766707f4fb7b8d218df448b03d4 P feb4b6069fa44de4adb52d343b9ce101: Deleting consensus metadata
I20260928 02:57:23.540516 15983 catalog_manager.cc:4841] TS feb4b6069fa44de4adb52d343b9ce101 (127.27.229.195:43023): tablet c3542766707f4fb7b8d218df448b03d4 (table test-table [id=ec08d0b10ac846ab923b103cbabc041e]) successfully deleted
I20260928 02:57:23.545094 16486 ts_tablet_manager.cc:1918] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 02:57:23.545321 16486 log.cc:1196] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Deleting WAL directory at /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790563915649057-28567-0/minicluster-data/ts-1/wal/wals/c3542766707f4fb7b8d218df448b03d4
I20260928 02:57:23.546060 16486 ts_tablet_manager.cc:1939] T c3542766707f4fb7b8d218df448b03d4 P 699dbd1a1f6546918cd3278c61d6b16c: Deleting consensus metadata
I20260928 02:57:23.548228 15982 catalog_manager.cc:4841] TS 699dbd1a1f6546918cd3278c61d6b16c (127.27.229.194:45505): tablet c3542766707f4fb7b8d218df448b03d4 (table test-table [id=ec08d0b10ac846ab923b103cbabc041e]) successfully deleted
I20260928 02:57:23.625005 16485 ts_tablet_manager.cc:1939] T c3542766707f4fb7b8d218df448b03d4 P 23b1728464204cb48c7fa68f924d02cb: Deleting consensus metadata
I20260928 02:57:23.625046 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16055
I20260928 02:57:23.643221 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16186
W20260928 02:57:23.643227 15982 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.229.193:38993 (error 108)
I20260928 02:57:23.645403 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.644899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56420)
0928 02:57:23.645080 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.645212 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:57:23.645704 15982 catalog_manager.cc:4584] TS 23b1728464204cb48c7fa68f924d02cb (127.27.229.193:38993): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3542766707f4fb7b8d218df448b03d4: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
I20260928 02:57:23.652041 16460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.644550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.27.229.194:45505)
0928 02:57:23.644838 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.644844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.644864 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:57:23.644936 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.644942 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.645056 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.645173 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:23.645686 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.646963 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.648259 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.648922 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.649072 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.650241 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:23.650253 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:23.650272 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:23.651177 (+ 905us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:23.651247 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:23.651272 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:23.651391 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.651544 (+ 153us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.651651 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260928 02:57:23.652900 16308 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:23.699487 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.698949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56434)
0928 02:57:23.699196 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.699294 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:57:23.700695 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.700280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56442)
0928 02:57:23.700471 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.700559 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 02:57:23.761678 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.760963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56458)
0928 02:57:23.761283 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.761446 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 02:57:23.762738 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.762387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56460)
0928 02:57:23.762553 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.762626 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 02:57:23.769881 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16186
I20260928 02:57:23.792984 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16316
I20260928 02:57:23.801723 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.793928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43492 (local address 127.27.229.195:43023)
0928 02:57:23.794134 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:23.794140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:23.794180 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:57:23.794244 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:23.794250 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:23.794350 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:23.794451 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:23.794824 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.795993 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.797899 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:23.798570 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:23.798705 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:23.800070 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:23.800081 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:23.800101 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:23.800887 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:23.800964 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:23.800990 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:23.801096 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:23.801250 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:57:23.801369 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:57:23.802485 16438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:23.843946 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.843348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56472)
0928 02:57:23.843573 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.843736 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 02:57:23.845074 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:23.844622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56474)
0928 02:57:23.844828 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:23.844935 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 02:57:23.896675 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16316
I20260928 02:57:23.913241 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 15965
I20260928 02:57:23.914377 16026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:24.026010 16459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:24.025337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.193:38993 (local address 127.0.0.1:56484)
0928 02:57:24.025585 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 02:57:24.025705 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.193:38993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 02:57:24.026358 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 15965
2026-09-28T02:57:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2787 ms)
[----------] 6 tests from EncryptionPolicyTest (22834 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:24 dist-test-slave-s1kg krb5kdc[16495](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:24 dist-test-slave-s1kg krb5kdc[16495](info): set up 2 sockets
Sep 28 02:57:24 dist-test-slave-s1kg krb5kdc[16495](info): commencing operation
krb5kdc: starting...
W20260928 02:57:25.143297 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:25 dist-test-slave-s1kg krb5kdc[16495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:25.422379 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:25.771390 16508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:25.771713 16508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:25.771867 16508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:25.779712 16508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:25.779842 16508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:25.779968 16508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:25.780095 16508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:25.780154 16508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:25.790623 16508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45101
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:25.792033 16508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:25.793285 16508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:25.805848 16513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:25.805859 16516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:25.806110 16514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:25.806762 16508 server_base.cc:1034] running on GCE node
I20260928 02:57:25.807722 16508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:25.809267 16508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:25.810525 16508 hybrid_clock.cc:648] HybridClock initialized: now 1790564245810478 us; error 49 us; skew 500 ppm
I20260928 02:57:25.811000 16508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:25 dist-test-slave-s1kg krb5kdc[16495](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:25.814545 16508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:25.816285 16508 webserver.cc:460] Webserver started at http://127.0.0.1:44541/ using document root <none> and password file <none>
I20260928 02:57:25.817004 16508 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:25.817152 16508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:25.817462 16508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:25.820662 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "217f363942274caeae6ecae1f6083abf"
format_stamp: "Formatted at 2026-09-28 02:57:25 on dist-test-slave-s1kg"
server_key: "7224e471c2e2032d8275a161de1d4978"
server_key_iv: "54e7c8432416a07108de58547f398447"
server_key_version: "encryptionkey@0"
I20260928 02:57:25.821491 16508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "217f363942274caeae6ecae1f6083abf"
format_stamp: "Formatted at 2026-09-28 02:57:25 on dist-test-slave-s1kg"
server_key: "7224e471c2e2032d8275a161de1d4978"
server_key_iv: "54e7c8432416a07108de58547f398447"
server_key_version: "encryptionkey@0"
I20260928 02:57:25.826922 16508 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260928 02:57:25.830813 16523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:25.832283 16508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:57:25.832509 16508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "217f363942274caeae6ecae1f6083abf"
format_stamp: "Formatted at 2026-09-28 02:57:25 on dist-test-slave-s1kg"
server_key: "7224e471c2e2032d8275a161de1d4978"
server_key_iv: "54e7c8432416a07108de58547f398447"
server_key_version: "encryptionkey@0"
I20260928 02:57:25.832719 16508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:25.883826 16508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:25.888713 16508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:25.889043 16508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:25.908613 16508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45101
I20260928 02:57:25.908653 16574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45101 every 8 connection(s)
I20260928 02:57:25.910104 16508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:25.913879 16575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:25.919515 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16508
I20260928 02:57:25.919744 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:25.920539 28567 external_mini_cluster.cc:1428] Setting key 580ece5be8c82907a85f8b4bf4376352
I20260928 02:57:25.925897 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf: Bootstrap starting.
I20260928 02:57:25.929816 16575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:25.931239 16575 log.cc:824] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:25.934944 16575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf: No bootstrap required, opened a new log
Sep 28 02:57:25 dist-test-slave-s1kg krb5kdc[16495](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:57:25.943588 16575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260928 02:57:25.943996 16575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:25.944130 16575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 217f363942274caeae6ecae1f6083abf, State: Initialized, Role: FOLLOWER
I20260928 02:57:25.944788 16575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260928 02:57:25.945052 16575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:25.945211 16575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:25.945396 16575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:25.946100 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:25.922810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45757 (local address 127.0.0.1:45101)
0928 02:57:25.923873 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:25.923886 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:25.923933 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:57:25.925648 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:25.925671 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:25.926130 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:25.926470 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:25.928428 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:25.929743 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:25.931637 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:25.932868 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:25.933025 (+ 157us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:25.938148 (+ 5123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:25.938156 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:25.938172 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:25.938221 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:25.941501 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:25.942334 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:25.942341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:25.942347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:25.942528 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:25.943019 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:25.943023 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:25.943025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:25.943549 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:25.943910 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:25.944396 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 02:57:25.944910 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":532,"threads_started":1}
I20260928 02:57:25.948421 16575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260928 02:57:25.949038 16575 leader_election.cc:302] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 217f363942274caeae6ecae1f6083abf; no voters:
I20260928 02:57:25.949738 16575 leader_election.cc:288] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:25.949877 16580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:25.951031 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:25.952459 16575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:25.956096 16580 raft_consensus.cc:686] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [term 1 LEADER]: Becoming Leader. State: Replica: 217f363942274caeae6ecae1f6083abf, State: Running, Role: LEADER
I20260928 02:57:25.957315 16580 consensus_queue.cc:237] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } }
I20260928 02:57:25.960807 16581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "217f363942274caeae6ecae1f6083abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } } }
I20260928 02:57:25.961345 16581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:25.962023 16582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 217f363942274caeae6ecae1f6083abf. Latest consensus state: current_term: 1 leader_uuid: "217f363942274caeae6ecae1f6083abf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "217f363942274caeae6ecae1f6083abf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45101 } } }
I20260928 02:57:25.962365 16582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:25.981794 16578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:25.956855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56165 (local address 127.0.0.1:45101)
0928 02:57:25.957069 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:25.957077 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:25.957505 (+ 428us) server_negotiation.cc:407] Connection header received
0928 02:57:25.957765 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:25.957772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:25.957870 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:25.957944 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:25.960170 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:25.961575 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:25.964223 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:25.965273 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:25.965423 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:25.970501 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:25.970514 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:25.970522 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:25.970587 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:25.976199 (+ 5612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:25.977752 (+ 1553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:25.977763 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:25.977770 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:25.977995 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:25.978798 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:25.978808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:25.978845 (+ 37us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:25.979479 (+ 634us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:25.979893 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:25.980899 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0928 02:57:25.981185 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":39}
I20260928 02:57:25.984455 16592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:25.989943 16592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:26.001390 16592 catalog_manager.cc:1347] Generated new cluster ID: 73822b7387a2452a8317e0469a1f2d5c
I20260928 02:57:26.001560 16592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:26.011423 16592 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:26.012912 16592 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:26.023522 16592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 217f363942274caeae6ecae1f6083abf: Generated new TSK 0
I20260928 02:57:26.024447 16592 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:26.037384 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16508
I20260928 02:57:26.040301 16570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:26.136543 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16508
W20260928 02:57:26.154516 16493 connection.cc:535] client connection to 127.0.0.1:45101 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45101): Connection reset by peer (error 104)
2026-09-28T02:57:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2105 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:26 dist-test-slave-s1kg krb5kdc[16608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:26 dist-test-slave-s1kg krb5kdc[16608](info): set up 2 sockets
Sep 28 02:57:26 dist-test-slave-s1kg krb5kdc[16608](info): commencing operation
krb5kdc: starting...
W20260928 02:57:27.267289 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:27 dist-test-slave-s1kg krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:27.535321 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:27.881948 16621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:27.882269 16621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:27.882480 16621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:27.889889 16621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:27.890014 16621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:27.890162 16621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:27.890280 16621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:27.890344 16621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:27.900579 16621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:27.902002 16621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:27.903401 16621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:27.916851 16626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:27.917016 16627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:27.916985 16629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:27.918102 16621 server_base.cc:1034] running on GCE node
I20260928 02:57:27.919034 16621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:27.920720 16621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:27.921972 16621 hybrid_clock.cc:648] HybridClock initialized: now 1790564247921916 us; error 50 us; skew 500 ppm
I20260928 02:57:27.922329 16621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:27 dist-test-slave-s1kg krb5kdc[16608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:27.930711 16621 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:27.932443 16621 webserver.cc:460] Webserver started at http://127.0.0.1:40385/ using document root <none> and password file <none>
I20260928 02:57:27.933153 16621 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:27.933259 16621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:27.933521 16621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:27.936614 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "6068344048674d9caaca3c69ea129fd8"
format_stamp: "Formatted at 2026-09-28 02:57:27 on dist-test-slave-s1kg"
server_key: "637857972f29f4baa249e5f44f398c8d"
server_key_iv: "f1c5e85d3e4f1e6bce1a94209623667a"
server_key_version: "encryptionkey@0"
I20260928 02:57:27.937336 16621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "6068344048674d9caaca3c69ea129fd8"
format_stamp: "Formatted at 2026-09-28 02:57:27 on dist-test-slave-s1kg"
server_key: "637857972f29f4baa249e5f44f398c8d"
server_key_iv: "f1c5e85d3e4f1e6bce1a94209623667a"
server_key_version: "encryptionkey@0"
I20260928 02:57:27.942564 16621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 02:57:27.946270 16636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:27.947820 16621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:27.948021 16621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "6068344048674d9caaca3c69ea129fd8"
format_stamp: "Formatted at 2026-09-28 02:57:27 on dist-test-slave-s1kg"
server_key: "637857972f29f4baa249e5f44f398c8d"
server_key_iv: "f1c5e85d3e4f1e6bce1a94209623667a"
server_key_version: "encryptionkey@0"
I20260928 02:57:27.948251 16621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:27.990657 16621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:27.995960 16621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:27.996304 16621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:28.017661 16621 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40053
I20260928 02:57:28.017707 16687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40053 every 8 connection(s)
I20260928 02:57:28.019192 16621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:28.021929 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16621
I20260928 02:57:28.022128 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:28.022585 28567 external_mini_cluster.cc:1428] Setting key 49527dbd0503de908863cfde6513a6a7
I20260928 02:57:28.023439 16688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:28.036331 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:28.024464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50901 (local address 127.0.0.1:40053)
0928 02:57:28.025280 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:28.025291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:28.025332 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:57:28.026980 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:28.027004 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:28.027409 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:28.027759 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:28.028390 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:28.029746 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:28.031618 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:28.032287 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:28.032417 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:28.033830 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:28.033837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:28.033852 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:28.034458 (+ 606us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:28.034538 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:28.034571 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:28.034715 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:28.035105 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 02:57:28.035306 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":391,"threads_started":1}
I20260928 02:57:28.037377 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8: Bootstrap starting.
I20260928 02:57:28.040200 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:28.040808 16688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:28.042032 16688 log.cc:824] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:28.045305 16688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8: No bootstrap required, opened a new log
I20260928 02:57:28.052331 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:28.045332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51355 (local address 127.0.0.1:40053)
0928 02:57:28.045518 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:28.045523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:28.045963 (+ 440us) server_negotiation.cc:407] Connection header received
0928 02:57:28.046124 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:28.046130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:28.046212 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:28.046336 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:28.046818 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:28.048049 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:28.049396 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:28.050062 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:28.050203 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:28.051484 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:28.051492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:28.051502 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:28.051525 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:28.051575 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:28.051596 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:28.051718 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:28.051891 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 02:57:28.051953 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:57:28.055277 16688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260928 02:57:28.055693 16688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:28.055838 16688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6068344048674d9caaca3c69ea129fd8, State: Initialized, Role: FOLLOWER
I20260928 02:57:28.056437 16688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260928 02:57:28.056670 16688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:28.056818 16688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:28.057030 16688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:28.059448 16688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260928 02:57:28.059945 16688 leader_election.cc:302] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6068344048674d9caaca3c69ea129fd8; no voters:
I20260928 02:57:28.060515 16688 leader_election.cc:288] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:28.060736 16698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:28.062505 16698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [term 1 LEADER]: Becoming Leader. State: Replica: 6068344048674d9caaca3c69ea129fd8, State: Running, Role: LEADER
I20260928 02:57:28.062628 16688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:28.063222 16698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } }
I20260928 02:57:28.065898 16700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6068344048674d9caaca3c69ea129fd8. Latest consensus state: current_term: 1 leader_uuid: "6068344048674d9caaca3c69ea129fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260928 02:57:28.066327 16700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:28.066952 16699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6068344048674d9caaca3c69ea129fd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6068344048674d9caaca3c69ea129fd8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40053 } } }
I20260928 02:57:28.067250 16699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:28.068379 16707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:28.074045 16707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:28.083556 16707 catalog_manager.cc:1347] Generated new cluster ID: 02a57d8b99e54b10b296b933617b317b
I20260928 02:57:28.083667 16707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:28.092661 16707 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:28.094005 16707 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:28.106523 16707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6068344048674d9caaca3c69ea129fd8: Generated new TSK 0
I20260928 02:57:28.107450 16707 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:28.115046 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16621
W20260928 02:57:28.116336 16683 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50901
W20260928 02:57:28.117157 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16621: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:28.117342 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16621
W20260928 02:57:28.134730 16606 connection.cc:535] client connection to 127.0.0.1:40053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40053): Connection reset by peer (error 104)
2026-09-28T02:57:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1980 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:28 dist-test-slave-s1kg krb5kdc[16719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:28 dist-test-slave-s1kg krb5kdc[16719](info): set up 2 sockets
Sep 28 02:57:28 dist-test-slave-s1kg krb5kdc[16719](info): commencing operation
krb5kdc: starting...
W20260928 02:57:29.240171 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:29 dist-test-slave-s1kg krb5kdc[16719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:29.518002 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:29.876302 16732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:29.876619 16732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:29.876775 16732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:29.884769 16732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:29.884908 16732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:29.884956 16732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:29.885077 16732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:29.885162 16732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:29.896274 16732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:29.897701 16732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:29.899103 16732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:29.912060 16737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:29.912173 16738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:29.912088 16740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:29.913697 16732 server_base.cc:1034] running on GCE node
I20260928 02:57:29.914518 16732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:29.916121 16732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:29.917448 16732 hybrid_clock.cc:648] HybridClock initialized: now 1790564249917388 us; error 46 us; skew 500 ppm
I20260928 02:57:29.917881 16732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:29 dist-test-slave-s1kg krb5kdc[16719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:29.921509 16732 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:29.923213 16732 webserver.cc:460] Webserver started at http://127.0.0.1:37441/ using document root <none> and password file <none>
I20260928 02:57:29.923954 16732 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:29.924103 16732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:29.924441 16732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:29.927697 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "b7a61fb76cc0490e90f32b6337cb3c8b"
format_stamp: "Formatted at 2026-09-28 02:57:29 on dist-test-slave-s1kg"
server_key: "7349da9544f528bb2ab7ddfeb52ca204"
server_key_iv: "1083c8322c3524988e5a4a409f97e754"
server_key_version: "encryptionkey@0"
I20260928 02:57:29.928560 16732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "b7a61fb76cc0490e90f32b6337cb3c8b"
format_stamp: "Formatted at 2026-09-28 02:57:29 on dist-test-slave-s1kg"
server_key: "7349da9544f528bb2ab7ddfeb52ca204"
server_key_iv: "1083c8322c3524988e5a4a409f97e754"
server_key_version: "encryptionkey@0"
I20260928 02:57:29.934279 16732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:57:29.938303 16747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:29.939771 16732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:29.939970 16732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "b7a61fb76cc0490e90f32b6337cb3c8b"
format_stamp: "Formatted at 2026-09-28 02:57:29 on dist-test-slave-s1kg"
server_key: "7349da9544f528bb2ab7ddfeb52ca204"
server_key_iv: "1083c8322c3524988e5a4a409f97e754"
server_key_version: "encryptionkey@0"
I20260928 02:57:29.940188 16732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:29.946521 16732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:29.951337 16732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:29.951668 16732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:29.972508 16732 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41313
I20260928 02:57:29.972566 16798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41313 every 8 connection(s)
I20260928 02:57:29.974042 16732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:29.974615 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16732
I20260928 02:57:29.974804 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:29.975468 28567 external_mini_cluster.cc:1428] Setting key 5963f0bf6edf0291009df7d49f06882e
I20260928 02:57:29.978541 16799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:29.983599 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:29.977410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56015 (local address 127.0.0.1:41313)
0928 02:57:29.978258 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:29.978270 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:29.978311 (+ 41us) server_negotiation.cc:407] Connection header received
0928 02:57:29.980056 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:29.980078 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:29.980403 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:29.980735 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:29.981874 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:29.981880 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:29.981897 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:29.981922 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:29.982003 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:29.982025 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:29.982201 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:29.982298 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 02:57:29.982591 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":368,"threads_started":1}
I20260928 02:57:29.987884 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:29.991626 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b: Bootstrap starting.
I20260928 02:57:29.995255 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:29.992523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55475 (local address 127.0.0.1:41313)
0928 02:57:29.992770 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:29.992780 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:29.993241 (+ 461us) server_negotiation.cc:407] Connection header received
0928 02:57:29.993463 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:29.993470 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:29.993528 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:29.993631 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:29.994557 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:29.994563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:29.994567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:29.994592 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:29.994634 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:29.994650 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:29.994769 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:29.994891 (+ 122us) server_negotiation.cc:299] Negotiation successful
0928 02:57:29.994957 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 02:57:29.995280 16799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:29.996838 16799 log.cc:824] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:29.999939 16799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b: No bootstrap required, opened a new log
I20260928 02:57:30.008119 16799 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } }
I20260928 02:57:30.008738 16799 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:30.008867 16799 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7a61fb76cc0490e90f32b6337cb3c8b, State: Initialized, Role: FOLLOWER
I20260928 02:57:30.009521 16799 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } }
I20260928 02:57:30.009774 16799 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:30.009927 16799 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:30.010121 16799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:30.012523 16799 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } }
I20260928 02:57:30.013077 16799 leader_election.cc:302] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7a61fb76cc0490e90f32b6337cb3c8b; no voters:
I20260928 02:57:30.013742 16799 leader_election.cc:288] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:30.013918 16809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:30.015101 16809 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [term 1 LEADER]: Becoming Leader. State: Replica: b7a61fb76cc0490e90f32b6337cb3c8b, State: Running, Role: LEADER
I20260928 02:57:30.015686 16809 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } }
I20260928 02:57:30.016083 16799 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:30.019219 16810 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } } }
I20260928 02:57:30.019198 16811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7a61fb76cc0490e90f32b6337cb3c8b. Latest consensus state: current_term: 1 leader_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7a61fb76cc0490e90f32b6337cb3c8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41313 } } }
I20260928 02:57:30.019798 16811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:30.019795 16810 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:30.028165 16817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:30.033563 16817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:30.043030 16817 catalog_manager.cc:1347] Generated new cluster ID: 791cfd7d6d9e4961af9b5475e8401960
I20260928 02:57:30.043170 16817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:30.065547 16817 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:30.066138 16817 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:30.086378 16817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7a61fb76cc0490e90f32b6337cb3c8b: Generated new TSK 0
I20260928 02:57:30.087574 16817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:30.100317 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16732
W20260928 02:57:30.101473 16794 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56015
W20260928 02:57:30.102253 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:30.102423 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16732
2026-09-28T02:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1987 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:30 dist-test-slave-s1kg krb5kdc[16830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:30 dist-test-slave-s1kg krb5kdc[16830](info): set up 2 sockets
Sep 28 02:57:30 dist-test-slave-s1kg krb5kdc[16830](info): commencing operation
krb5kdc: starting...
W20260928 02:57:31.231249 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:31 dist-test-slave-s1kg krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:31.506949 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38903
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39515
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:31.856396 16843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:31.856700 16843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:31.856879 16843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:31.864409 16843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:31.864537 16843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:31.864583 16843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:31.864699 16843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:31.864763 16843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:31.875164 16843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39515
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:31.876609 16843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:31.878002 16843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:31.890818 16848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:31.890856 16849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:31.890936 16851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:31.891953 16843 server_base.cc:1034] running on GCE node
I20260928 02:57:31.892838 16843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:31.894470 16843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:31.895723 16843 hybrid_clock.cc:648] HybridClock initialized: now 1790564251895671 us; error 47 us; skew 500 ppm
I20260928 02:57:31.896126 16843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:31 dist-test-slave-s1kg krb5kdc[16830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:31.899511 16843 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:31.901229 16843 webserver.cc:460] Webserver started at http://127.0.0.1:45371/ using document root <none> and password file <none>
I20260928 02:57:31.901968 16843 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:31.902123 16843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:31.902448 16843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:31.905815 16843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "beee301f0f9a446b92e5d9507c725217"
format_stamp: "Formatted at 2026-09-28 02:57:31 on dist-test-slave-s1kg"
server_key: "7d290c9b51d0d265e5948d718d1db2da"
server_key_iv: "730cf02f665c81fe269a72d64ad1850b"
server_key_version: "encryptionkey@0"
I20260928 02:57:31.906661 16843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "beee301f0f9a446b92e5d9507c725217"
format_stamp: "Formatted at 2026-09-28 02:57:31 on dist-test-slave-s1kg"
server_key: "7d290c9b51d0d265e5948d718d1db2da"
server_key_iv: "730cf02f665c81fe269a72d64ad1850b"
server_key_version: "encryptionkey@0"
I20260928 02:57:31.912238 16843 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260928 02:57:31.916003 16858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:31.917371 16843 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:31.917578 16843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "beee301f0f9a446b92e5d9507c725217"
format_stamp: "Formatted at 2026-09-28 02:57:31 on dist-test-slave-s1kg"
server_key: "7d290c9b51d0d265e5948d718d1db2da"
server_key_iv: "730cf02f665c81fe269a72d64ad1850b"
server_key_version: "encryptionkey@0"
I20260928 02:57:31.917800 16843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:31.936493 16843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:31.941390 16843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:31.941713 16843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:31.961659 16843 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38903
I20260928 02:57:31.961700 16909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38903 every 8 connection(s)
I20260928 02:57:31.963213 16843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:31.965119 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16843
I20260928 02:57:31.965305 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:31.965840 28567 external_mini_cluster.cc:1428] Setting key 570326b17bfaf84fcfbea75ba73798f0
I20260928 02:57:31.968113 16910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:57:31 dist-test-slave-s1kg krb5kdc[16830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:57:31.982082 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217: Bootstrap starting.
I20260928 02:57:31.985561 16910 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:31.986655 16910 log.cc:824] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:31.989931 16910 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217: No bootstrap required, opened a new log
I20260928 02:57:31.989956 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:31.967734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34411 (local address 127.0.0.1:38903)
0928 02:57:31.968571 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:31.968584 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:31.968626 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:57:31.970248 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:31.970270 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:31.970687 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:31.971127 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:31.973032 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:31.974397 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:31.976290 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:31.977012 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:31.977163 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:31.982270 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:31.982285 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:31.982307 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:31.982359 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:31.985574 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:31.986360 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:31.986367 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:31.986374 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:31.986552 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:31.987005 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:31.987008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:31.987010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:31.987782 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:31.988126 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:31.988525 (+ 399us) server_negotiation.cc:299] Negotiation successful
0928 02:57:31.988902 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":376,"threads_started":1}
I20260928 02:57:31.994349 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:31.998611 16910 raft_consensus.cc:348] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } }
I20260928 02:57:31.999253 16910 raft_consensus.cc:374] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:31.999403 16910 raft_consensus.cc:729] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beee301f0f9a446b92e5d9507c725217, State: Initialized, Role: FOLLOWER
I20260928 02:57:32.000161 16910 consensus_queue.cc:260] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } }
I20260928 02:57:32.000496 16910 raft_consensus.cc:388] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:32.000653 16910 raft_consensus.cc:482] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:32.000850 16910 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:32.004030 16910 raft_consensus.cc:504] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } }
I20260928 02:57:32.004781 16910 leader_election.cc:302] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beee301f0f9a446b92e5d9507c725217; no voters:
I20260928 02:57:32.005576 16910 leader_election.cc:288] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:32.005707 16920 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:32.009109 16910 sys_catalog.cc:564] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:32.011157 16920 raft_consensus.cc:686] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [term 1 LEADER]: Becoming Leader. State: Replica: beee301f0f9a446b92e5d9507c725217, State: Running, Role: LEADER
I20260928 02:57:32.012135 16920 consensus_queue.cc:237] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } }
I20260928 02:57:32.016239 16913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:31.999000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43507 (local address 127.0.0.1:38903)
0928 02:57:31.999281 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:31.999287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:31.999864 (+ 577us) server_negotiation.cc:407] Connection header received
0928 02:57:32.000073 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:32.000077 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:32.000146 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:32.000240 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:32.002172 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:32.003081 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:32.004908 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:32.005451 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:32.005640 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:32.008500 (+ 2860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:32.008506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:32.008509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:32.008549 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:32.012020 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:32.012916 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:32.012923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:32.012926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:32.013121 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:32.013515 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:32.013520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:32.013524 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:32.013848 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:32.015419 (+ 1571us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:32.015602 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 02:57:32.015856 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260928 02:57:32.016314 16921 sys_catalog.cc:455] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beee301f0f9a446b92e5d9507c725217. Latest consensus state: current_term: 1 leader_uuid: "beee301f0f9a446b92e5d9507c725217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } } }
I20260928 02:57:32.017966 16921 sys_catalog.cc:458] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:32.017590 16922 sys_catalog.cc:455] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beee301f0f9a446b92e5d9507c725217" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beee301f0f9a446b92e5d9507c725217" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38903 } } }
I20260928 02:57:32.019789 16922 sys_catalog.cc:458] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:32.023830 16926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:32.030490 16926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:32.040913 16926 catalog_manager.cc:1347] Generated new cluster ID: 561b48dee4a64ddb81d50a498992ce22
I20260928 02:57:32.041070 16926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:32.055683 16926 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:32.057078 16926 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:32.065959 16926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beee301f0f9a446b92e5d9507c725217: Generated new TSK 0
I20260928 02:57:32.067193 16926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:32.079017 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16843
I20260928 02:57:32.080204 16905 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:32.172906 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16843
2026-09-28T02:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2066 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:32 dist-test-slave-s1kg krb5kdc[16942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:32 dist-test-slave-s1kg krb5kdc[16942](info): set up 2 sockets
Sep 28 02:57:32 dist-test-slave-s1kg krb5kdc[16942](info): commencing operation
krb5kdc: starting...
W20260928 02:57:33.291191 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:33 dist-test-slave-s1kg krb5kdc[16942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:33.574692 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45023
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:33.919876 16955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:33.920246 16955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:33.920423 16955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:33.928615 16955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:33.928813 16955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:33.928928 16955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:33.929051 16955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:33.929131 16955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:33.939610 16955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:33.941041 16955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:33.942451 16955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:33.955482 16960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:33.955786 16963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:33.955961 16961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:33.956637 16955 server_base.cc:1034] running on GCE node
I20260928 02:57:33.957544 16955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:33.959195 16955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:33.960424 16955 hybrid_clock.cc:648] HybridClock initialized: now 1790564253960395 us; error 64 us; skew 500 ppm
I20260928 02:57:33.960821 16955 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:33 dist-test-slave-s1kg krb5kdc[16942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:33.967835 16955 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:33.969551 16955 webserver.cc:460] Webserver started at http://127.0.0.1:42469/ using document root <none> and password file <none>
I20260928 02:57:33.970286 16955 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:33.970407 16955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:33.970695 16955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:33.974176 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "c62ece797b194346a157efb2dac18141"
format_stamp: "Formatted at 2026-09-28 02:57:33 on dist-test-slave-s1kg"
server_key: "da1fe8bdfea7f14ad46b5dc520ba7a0a"
server_key_iv: "e73a216212d929ac50b1da65aecebbcd"
server_key_version: "encryptionkey@0"
I20260928 02:57:33.975072 16955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "c62ece797b194346a157efb2dac18141"
format_stamp: "Formatted at 2026-09-28 02:57:33 on dist-test-slave-s1kg"
server_key: "da1fe8bdfea7f14ad46b5dc520ba7a0a"
server_key_iv: "e73a216212d929ac50b1da65aecebbcd"
server_key_version: "encryptionkey@0"
I20260928 02:57:33.980764 16955 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 02:57:33.984557 16970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:33.986212 16955 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 02:57:33.986438 16955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "c62ece797b194346a157efb2dac18141"
format_stamp: "Formatted at 2026-09-28 02:57:33 on dist-test-slave-s1kg"
server_key: "da1fe8bdfea7f14ad46b5dc520ba7a0a"
server_key_iv: "e73a216212d929ac50b1da65aecebbcd"
server_key_version: "encryptionkey@0"
I20260928 02:57:33.986655 16955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:34.003674 16955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:34.008391 16955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:34.008703 16955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:34.028487 16955 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45023
I20260928 02:57:34.028546 17021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45023 every 8 connection(s)
I20260928 02:57:34.029928 16955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:34.032311 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 16955
I20260928 02:57:34.032507 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:34.033056 28567 external_mini_cluster.cc:1428] Setting key f035c297d48ddb60fe4177ef0a905020
I20260928 02:57:34.034718 17022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:34.047382 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:34.035006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53573 (local address 127.0.0.1:45023)
0928 02:57:34.035849 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:34.035860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:34.035902 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:57:34.037544 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:34.037568 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:34.037994 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:34.038358 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:34.039011 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:34.040428 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:34.042304 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:34.043123 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:34.043263 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:34.044637 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:34.044647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:34.044666 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:34.045293 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:34.045399 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:34.045431 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:34.045540 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:34.045849 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 02:57:34.046219 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":387,"threads_started":1}
I20260928 02:57:34.048511 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141: Bootstrap starting.
I20260928 02:57:34.051450 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:34.052038 17022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:34.053314 17022 log.cc:824] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:34.056639 17022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141: No bootstrap required, opened a new log
I20260928 02:57:34.066032 17025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:34.058718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38337 (local address 127.0.0.1:45023)
0928 02:57:34.059060 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:34.059073 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:34.059454 (+ 381us) server_negotiation.cc:407] Connection header received
0928 02:57:34.059681 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:34.059686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:34.059779 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:34.059883 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:34.060540 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:34.061314 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:34.063373 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:34.063854 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:34.063966 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:34.065221 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:34.065226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:34.065230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:34.065253 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:34.065300 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:34.065315 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:34.065396 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:34.065553 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 02:57:34.065643 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 02:57:34.067211 17022 raft_consensus.cc:348] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260928 02:57:34.067739 17022 raft_consensus.cc:374] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:34.067854 17022 raft_consensus.cc:729] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c62ece797b194346a157efb2dac18141, State: Initialized, Role: FOLLOWER
I20260928 02:57:34.068414 17022 consensus_queue.cc:260] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260928 02:57:34.068629 17022 raft_consensus.cc:388] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:34.068732 17022 raft_consensus.cc:482] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:34.068863 17022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:34.071601 17022 raft_consensus.cc:504] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260928 02:57:34.072229 17022 leader_election.cc:302] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c62ece797b194346a157efb2dac18141; no voters:
I20260928 02:57:34.072830 17022 leader_election.cc:288] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:34.073122 17032 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:34.073884 17032 raft_consensus.cc:686] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [term 1 LEADER]: Becoming Leader. State: Replica: c62ece797b194346a157efb2dac18141, State: Running, Role: LEADER
I20260928 02:57:34.074437 17032 consensus_queue.cc:237] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } }
I20260928 02:57:34.074995 17022 sys_catalog.cc:564] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:34.077769 17033 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c62ece797b194346a157efb2dac18141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } } }
I20260928 02:57:34.078181 17033 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:34.079648 17034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c62ece797b194346a157efb2dac18141. Latest consensus state: current_term: 1 leader_uuid: "c62ece797b194346a157efb2dac18141" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c62ece797b194346a157efb2dac18141" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45023 } } }
I20260928 02:57:34.080148 17034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:34.086381 17037 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:34.091166 17037 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:34.099920 17037 catalog_manager.cc:1347] Generated new cluster ID: 015626c8824a48d3b8c1296aec62921e
I20260928 02:57:34.100063 17037 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:34.115948 17037 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:34.117362 17037 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:34.129652 17037 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c62ece797b194346a157efb2dac18141: Generated new TSK 0
I20260928 02:57:34.130631 17037 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:34.142122 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16955
W20260928 02:57:34.143538 17017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53573
W20260928 02:57:34.144323 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 16955: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:34.144508 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 16955
2026-09-28T02:57:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:34 dist-test-slave-s1kg krb5kdc[17053](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:34 dist-test-slave-s1kg krb5kdc[17053](info): set up 2 sockets
Sep 28 02:57:34 dist-test-slave-s1kg krb5kdc[17053](info): commencing operation
krb5kdc: starting...
W20260928 02:57:35.275324 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:35 dist-test-slave-s1kg krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:35.551107 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40811
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40811
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:35.887773 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:35.888140 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:35.888305 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:35.895870 17066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:35.896032 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:35.896145 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:35.896251 17066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:35.896328 17066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:35.907189 17066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40811
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40811
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:35.908700 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:35.910115 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:35.923760 17072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:35.923825 17071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:35.923828 17074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:35.924490 17066 server_base.cc:1034] running on GCE node
I20260928 02:57:35.925362 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:35.927089 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:35.928349 17066 hybrid_clock.cc:648] HybridClock initialized: now 1790564255928284 us; error 57 us; skew 500 ppm
I20260928 02:57:35.928768 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:35 dist-test-slave-s1kg krb5kdc[17053](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:35.932431 17066 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:35.934188 17066 webserver.cc:460] Webserver started at http://127.0.0.1:45905/ using document root <none> and password file <none>
I20260928 02:57:35.935029 17066 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:35.935200 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:35.935523 17066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:35.938758 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "8a4a323bc5144bb9a4c294322d13bc96"
format_stamp: "Formatted at 2026-09-28 02:57:35 on dist-test-slave-s1kg"
server_key: "9506abfb84465cab5ef6d5bfba84f67e"
server_key_iv: "c4088c816aba56a04df805d090ed0be4"
server_key_version: "encryptionkey@0"
I20260928 02:57:35.939703 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "8a4a323bc5144bb9a4c294322d13bc96"
format_stamp: "Formatted at 2026-09-28 02:57:35 on dist-test-slave-s1kg"
server_key: "9506abfb84465cab5ef6d5bfba84f67e"
server_key_iv: "c4088c816aba56a04df805d090ed0be4"
server_key_version: "encryptionkey@0"
I20260928 02:57:35.945701 17066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260928 02:57:35.949707 17081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:35.951349 17066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:35.951573 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "8a4a323bc5144bb9a4c294322d13bc96"
format_stamp: "Formatted at 2026-09-28 02:57:35 on dist-test-slave-s1kg"
server_key: "9506abfb84465cab5ef6d5bfba84f67e"
server_key_iv: "c4088c816aba56a04df805d090ed0be4"
server_key_version: "encryptionkey@0"
I20260928 02:57:35.951809 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:35.958386 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:35.963222 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:35.963541 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:35.983729 17066 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40811
I20260928 02:57:35.983822 17132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40811 every 8 connection(s)
I20260928 02:57:35.985273 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:35.987106 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17066
I20260928 02:57:35.987284 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:35.987844 28567 external_mini_cluster.cc:1428] Setting key bf2c81d1ae6c768174dcff9590aedc54
I20260928 02:57:35.990033 17133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:35.996142 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:35.989789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42469 (local address 127.0.0.1:40811)
0928 02:57:35.990645 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:35.990657 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:35.990702 (+ 45us) server_negotiation.cc:407] Connection header received
0928 02:57:35.992406 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:35.992428 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:35.992807 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:35.993163 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:35.994297 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:35.994304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:35.994322 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:35.994337 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:35.994418 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:35.994445 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:35.994715 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:35.994890 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 02:57:35.995128 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":403,"threads_started":1}
I20260928 02:57:36.000296 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:36.003022 17133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96: Bootstrap starting.
I20260928 02:57:36.007294 17133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:36.007694 17136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:36.004841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33113 (local address 127.0.0.1:40811)
0928 02:57:36.005059 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:36.005066 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:36.005436 (+ 370us) server_negotiation.cc:407] Connection header received
0928 02:57:36.005577 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:36.005581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:36.005640 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:36.005734 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:36.006792 (+ 1058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:36.006796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:36.006799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:36.006815 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:36.006927 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:36.006947 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:36.007066 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:36.007215 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 02:57:36.007271 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 02:57:36.008952 17133 log.cc:824] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:36.012334 17133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96: No bootstrap required, opened a new log
I20260928 02:57:36.020509 17133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } }
I20260928 02:57:36.021027 17133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:36.021142 17133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4a323bc5144bb9a4c294322d13bc96, State: Initialized, Role: FOLLOWER
I20260928 02:57:36.021755 17133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } }
I20260928 02:57:36.022019 17133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:36.022182 17133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:36.022372 17133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:36.024757 17133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } }
I20260928 02:57:36.025363 17133 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a4a323bc5144bb9a4c294322d13bc96; no voters:
I20260928 02:57:36.026053 17133 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:36.026244 17143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:36.027464 17143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [term 1 LEADER]: Becoming Leader. State: Replica: 8a4a323bc5144bb9a4c294322d13bc96, State: Running, Role: LEADER
I20260928 02:57:36.028052 17143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } }
I20260928 02:57:36.028378 17133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:36.031318 17144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a4a323bc5144bb9a4c294322d13bc96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } } }
I20260928 02:57:36.031234 17145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4a323bc5144bb9a4c294322d13bc96. Latest consensus state: current_term: 1 leader_uuid: "8a4a323bc5144bb9a4c294322d13bc96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a4a323bc5144bb9a4c294322d13bc96" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40811 } } }
I20260928 02:57:36.031731 17144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:36.031752 17145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:36.035017 17151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:36.040691 17151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:36.050988 17151 catalog_manager.cc:1347] Generated new cluster ID: d21ac74499fc4dd0ac96db51c8fa5f10
I20260928 02:57:36.051142 17151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:36.075655 17151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:36.076247 17151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:36.085717 17151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4a323bc5144bb9a4c294322d13bc96: Generated new TSK 0
I20260928 02:57:36.086978 17151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:36.102243 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17066
W20260928 02:57:36.103636 17128 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42469
W20260928 02:57:36.104439 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:36.104609 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17066
2026-09-28T02:57:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1960 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:36 dist-test-slave-s1kg krb5kdc[17164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:36 dist-test-slave-s1kg krb5kdc[17164](info): set up 2 sockets
Sep 28 02:57:36 dist-test-slave-s1kg krb5kdc[17164](info): commencing operation
krb5kdc: starting...
W20260928 02:57:37.307303 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.165s user 0.004s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:37 dist-test-slave-s1kg krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:37.579218 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39361
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39361
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:37.918467 17177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:37.918752 17177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:37.918939 17177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:37.926426 17177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:37.926553 17177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:37.926657 17177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:37.926769 17177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:37.926882 17177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:37.937045 17177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39361
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39361
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:37.938407 17177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:37.939754 17177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:37.951915 17182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:37.951905 17183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:37.951911 17185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:37.952978 17177 server_base.cc:1034] running on GCE node
I20260928 02:57:37.953929 17177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:37.955485 17177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:37.956698 17177 hybrid_clock.cc:648] HybridClock initialized: now 1790564257956662 us; error 28 us; skew 500 ppm
I20260928 02:57:37.957027 17177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:37 dist-test-slave-s1kg krb5kdc[17164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564257, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:37.960248 17177 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:37.961800 17177 webserver.cc:460] Webserver started at http://127.0.0.1:46213/ using document root <none> and password file <none>
I20260928 02:57:37.962505 17177 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:37.962631 17177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:37.963006 17177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:37.966089 17177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "b611ec29cc154daaba855d9db0f30e8b"
format_stamp: "Formatted at 2026-09-28 02:57:37 on dist-test-slave-s1kg"
server_key: "07da5787caf7cc82d7c6a5dd3931de1c"
server_key_iv: "567517592e4c4a2ec3eadb3366f3ab1f"
server_key_version: "encryptionkey@0"
I20260928 02:57:37.966930 17177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "b611ec29cc154daaba855d9db0f30e8b"
format_stamp: "Formatted at 2026-09-28 02:57:37 on dist-test-slave-s1kg"
server_key: "07da5787caf7cc82d7c6a5dd3931de1c"
server_key_iv: "567517592e4c4a2ec3eadb3366f3ab1f"
server_key_version: "encryptionkey@0"
I20260928 02:57:37.972256 17177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:57:37.976328 17192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:37.977977 17177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:37.978178 17177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "b611ec29cc154daaba855d9db0f30e8b"
format_stamp: "Formatted at 2026-09-28 02:57:37 on dist-test-slave-s1kg"
server_key: "07da5787caf7cc82d7c6a5dd3931de1c"
server_key_iv: "567517592e4c4a2ec3eadb3366f3ab1f"
server_key_version: "encryptionkey@0"
I20260928 02:57:37.978437 17177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:38.004660 17177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:38.009353 17177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:38.009663 17177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:38.029072 17177 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39361
I20260928 02:57:38.029110 17243 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39361 every 8 connection(s)
I20260928 02:57:38.030508 17177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:38.034294 17244 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:38.036037 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17177
I20260928 02:57:38.036223 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:38.036736 28567 external_mini_cluster.cc:1428] Setting key 2df07dade0dde6a8fdec8ff7131bf436
I20260928 02:57:38.049139 17244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b: Bootstrap starting.
Sep 28 02:57:38 dist-test-slave-s1kg krb5kdc[17164](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:57:38.052829 17244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:38.053951 17244 log.cc:824] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:38.057197 17244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b: No bootstrap required, opened a new log
I20260928 02:57:38.061162 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:38.038539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:51931 (local address 127.0.0.1:39361)
0928 02:57:38.039406 (+ 867us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:38.039421 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:38.039468 (+ 47us) server_negotiation.cc:407] Connection header received
0928 02:57:38.041064 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:38.041086 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:38.041502 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:38.041836 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:38.043587 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:38.044991 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:38.046906 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:38.047653 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:38.047792 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:38.052504 (+ 4712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:38.052513 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:38.052529 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:38.052570 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:38.056330 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:38.057276 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:38.057283 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:38.057292 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:38.057497 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:38.057969 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:38.057973 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:38.057976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:38.058643 (+ 667us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:38.059010 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:38.059410 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 02:57:38.059765 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":429,"threads_started":1}
I20260928 02:57:38.065397 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:38.067099 17244 raft_consensus.cc:348] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } }
I20260928 02:57:38.067487 17244 raft_consensus.cc:374] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:38.067587 17244 raft_consensus.cc:729] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b611ec29cc154daaba855d9db0f30e8b, State: Initialized, Role: FOLLOWER
I20260928 02:57:38.068256 17244 consensus_queue.cc:260] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } }
I20260928 02:57:38.068588 17244 raft_consensus.cc:388] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:38.068750 17244 raft_consensus.cc:482] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:38.068929 17244 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:38.071745 17244 raft_consensus.cc:504] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } }
I20260928 02:57:38.072264 17244 leader_election.cc:302] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b611ec29cc154daaba855d9db0f30e8b; no voters:
I20260928 02:57:38.072968 17244 leader_election.cc:288] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:38.073082 17254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:38.075137 17254 raft_consensus.cc:686] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [term 1 LEADER]: Becoming Leader. State: Replica: b611ec29cc154daaba855d9db0f30e8b, State: Running, Role: LEADER
I20260928 02:57:38.075249 17244 sys_catalog.cc:564] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:38.075728 17254 consensus_queue.cc:237] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } }
I20260928 02:57:38.078653 17255 sys_catalog.cc:455] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b611ec29cc154daaba855d9db0f30e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } } }
I20260928 02:57:38.079114 17255 sys_catalog.cc:458] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:38.083674 17256 sys_catalog.cc:455] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b611ec29cc154daaba855d9db0f30e8b. Latest consensus state: current_term: 1 leader_uuid: "b611ec29cc154daaba855d9db0f30e8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b611ec29cc154daaba855d9db0f30e8b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39361 } } }
I20260928 02:57:38.084012 17256 sys_catalog.cc:458] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:38.084326 17259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:38.088387 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:38.069915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:40133 (local address 127.0.0.1:39361)
0928 02:57:38.070103 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:38.070109 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:38.070624 (+ 515us) server_negotiation.cc:407] Connection header received
0928 02:57:38.070739 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:38.070744 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:38.070907 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:38.071015 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:38.072404 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:38.073694 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:38.075491 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:38.076196 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:38.076337 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:38.080140 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:38.080151 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:38.080159 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:38.080214 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:38.084695 (+ 4481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:38.085483 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:38.085488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:38.085492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:38.085626 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:38.086121 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:38.086126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:38.086129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:38.086464 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:38.086760 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:38.087780 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0928 02:57:38.088005 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 02:57:38.091265 17259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:38.101204 17259 catalog_manager.cc:1347] Generated new cluster ID: 8ee9113ef5a24306894a0c77f7415fdb
I20260928 02:57:38.101332 17259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:38.126456 17259 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:38.127856 17259 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:38.133821 17259 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b611ec29cc154daaba855d9db0f30e8b: Generated new TSK 0
I20260928 02:57:38.134681 17259 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:38.147787 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17177
I20260928 02:57:38.149147 17239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:38.241087 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17177
W20260928 02:57:38.259019 17162 connection.cc:535] client connection to 127.0.0.1:39361 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39361): Connection reset by peer (error 104)
2026-09-28T02:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2137 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:38 dist-test-slave-s1kg krb5kdc[17276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:38 dist-test-slave-s1kg krb5kdc[17276](info): set up 2 sockets
Sep 28 02:57:38 dist-test-slave-s1kg krb5kdc[17276](info): commencing operation
krb5kdc: starting...
W20260928 02:57:39.387405 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.109s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:39 dist-test-slave-s1kg krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:39.658387 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:40.016557 17289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:40.016947 17289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:40.017129 17289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:40.024611 17289 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:40.024780 17289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:40.024890 17289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:40.024998 17289 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:40.025063 17289 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:40.035352 17289 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:40.036725 17289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:40.038033 17289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:40.050256 17297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:40.050334 17294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:40.050315 17295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:40.051122 17289 server_base.cc:1034] running on GCE node
I20260928 02:57:40.051955 17289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:40.053439 17289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:40.054716 17289 hybrid_clock.cc:648] HybridClock initialized: now 1790564260054650 us; error 61 us; skew 500 ppm
I20260928 02:57:40.055131 17289 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:40 dist-test-slave-s1kg krb5kdc[17276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:40.058720 17289 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:40.060467 17289 webserver.cc:460] Webserver started at http://127.0.0.1:38499/ using document root <none> and password file <none>
I20260928 02:57:40.061220 17289 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:40.061375 17289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:40.061698 17289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:40.065006 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "9b59c085f6d44b218035b06bddc373da"
format_stamp: "Formatted at 2026-09-28 02:57:40 on dist-test-slave-s1kg"
server_key: "653b2c4dbfcda5e37fe6728b1d397d56"
server_key_iv: "1c4c22c1c786f71f323f027b82100bd9"
server_key_version: "encryptionkey@0"
I20260928 02:57:40.065841 17289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "9b59c085f6d44b218035b06bddc373da"
format_stamp: "Formatted at 2026-09-28 02:57:40 on dist-test-slave-s1kg"
server_key: "653b2c4dbfcda5e37fe6728b1d397d56"
server_key_iv: "1c4c22c1c786f71f323f027b82100bd9"
server_key_version: "encryptionkey@0"
I20260928 02:57:40.071504 17289 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:57:40.075273 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:40.076743 17289 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260928 02:57:40.076983 17289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "9b59c085f6d44b218035b06bddc373da"
format_stamp: "Formatted at 2026-09-28 02:57:40 on dist-test-slave-s1kg"
server_key: "653b2c4dbfcda5e37fe6728b1d397d56"
server_key_iv: "1c4c22c1c786f71f323f027b82100bd9"
server_key_version: "encryptionkey@0"
I20260928 02:57:40.077232 17289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:40.095937 17289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:40.100750 17289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:40.101089 17289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:40.122742 17289 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43973
I20260928 02:57:40.122768 17355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43973 every 8 connection(s)
I20260928 02:57:40.124541 17289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:40.125445 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17289
I20260928 02:57:40.125685 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:40.126261 28567 external_mini_cluster.cc:1428] Setting key 4f11066795e78fc955cc58a13713577c
I20260928 02:57:40.129419 17356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:40.139927 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:40.128262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:47465 (local address 127.0.0.1:43973)
0928 02:57:40.129091 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:40.129103 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:40.129145 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:57:40.131045 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:40.131068 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:40.131476 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:40.131915 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:40.132486 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:40.133810 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:40.135349 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:40.136027 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:40.136227 (+ 200us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:40.137561 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:40.137568 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:40.137613 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:40.138188 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:40.138263 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:40.138288 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:40.138463 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:40.138740 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 02:57:40.138964 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":373,"threads_started":1}
I20260928 02:57:40.140838 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da: Bootstrap starting.
I20260928 02:57:40.144366 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:40.144783 17356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:40.145844 17356 log.cc:824] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:40.149571 17356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da: No bootstrap required, opened a new log
I20260928 02:57:40.155794 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:40.149083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:35889 (local address 127.0.0.1:43973)
0928 02:57:40.149249 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:40.149253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:40.149777 (+ 524us) server_negotiation.cc:407] Connection header received
0928 02:57:40.150021 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:40.150026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:40.150095 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:40.150216 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:40.150738 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:40.151592 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:40.152938 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:40.153395 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:40.153568 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:40.154944 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:40.154953 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:40.154960 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:40.154986 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:40.155043 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:40.155061 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:40.155211 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:40.155406 (+ 195us) server_negotiation.cc:299] Negotiation successful
0928 02:57:40.155470 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 02:57:40.159307 17356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260928 02:57:40.159775 17356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:40.159907 17356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b59c085f6d44b218035b06bddc373da, State: Initialized, Role: FOLLOWER
I20260928 02:57:40.160627 17356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260928 02:57:40.160907 17356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:40.161073 17356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:40.161304 17356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:40.164613 17356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260928 02:57:40.165192 17356 leader_election.cc:302] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b59c085f6d44b218035b06bddc373da; no voters:
I20260928 02:57:40.165807 17356 leader_election.cc:288] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:40.166023 17366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:40.166931 17366 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [term 1 LEADER]: Becoming Leader. State: Replica: 9b59c085f6d44b218035b06bddc373da, State: Running, Role: LEADER
I20260928 02:57:40.167529 17366 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } }
I20260928 02:57:40.168154 17356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:40.170562 17368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b59c085f6d44b218035b06bddc373da. Latest consensus state: current_term: 1 leader_uuid: "9b59c085f6d44b218035b06bddc373da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260928 02:57:40.170990 17368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:40.175525 17367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b59c085f6d44b218035b06bddc373da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b59c085f6d44b218035b06bddc373da" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43973 } } }
I20260928 02:57:40.175983 17367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:40.177520 17372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:40.183740 17372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:40.193460 17372 catalog_manager.cc:1347] Generated new cluster ID: 4ff1608ad8e6423d969905f629b3927f
I20260928 02:57:40.193595 17372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:40.221058 17372 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:40.222470 17372 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:40.231503 17372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b59c085f6d44b218035b06bddc373da: Generated new TSK 0
I20260928 02:57:40.232630 17372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:40.244189 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17289
W20260928 02:57:40.245934 17351 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:47465
W20260928 02:57:40.246934 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17289: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:40.247105 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17289
W20260928 02:57:40.264914 17274 connection.cc:535] client connection to 127.0.0.1:43973 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43973): Connection reset by peer (error 104)
2026-09-28T02:57:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2005 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:40 dist-test-slave-s1kg krb5kdc[17387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:40 dist-test-slave-s1kg krb5kdc[17387](info): set up 2 sockets
Sep 28 02:57:40 dist-test-slave-s1kg krb5kdc[17387](info): commencing operation
krb5kdc: starting...
W20260928 02:57:41.365111 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:41 dist-test-slave-s1kg krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:41.643328 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:41.994096 17400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:41.994398 17400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:41.994565 17400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:42.002897 17400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:42.003031 17400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:42.003114 17400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:42.003240 17400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:42.003304 17400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:42.013759 17400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:42.015192 17400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:42.016535 17400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:42.029336 17405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:42.029307 17408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:42.029289 17406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:42.030984 17400 server_base.cc:1034] running on GCE node
I20260928 02:57:42.031859 17400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:42.033213 17400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:42.034415 17400 hybrid_clock.cc:648] HybridClock initialized: now 1790564262034382 us; error 29 us; skew 500 ppm
I20260928 02:57:42.034780 17400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:42 dist-test-slave-s1kg krb5kdc[17387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:42.037881 17400 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:42.039458 17400 webserver.cc:460] Webserver started at http://127.0.0.1:40409/ using document root <none> and password file <none>
I20260928 02:57:42.040184 17400 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:42.040330 17400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:42.040630 17400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:42.043690 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "08f2ff0521d143698ce143139fdf202f"
format_stamp: "Formatted at 2026-09-28 02:57:42 on dist-test-slave-s1kg"
server_key: "ba6e9035c2f7617a5ba2252527be358f"
server_key_iv: "d58490eab966629d67e2a376194d581d"
server_key_version: "encryptionkey@0"
I20260928 02:57:42.044507 17400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "08f2ff0521d143698ce143139fdf202f"
format_stamp: "Formatted at 2026-09-28 02:57:42 on dist-test-slave-s1kg"
server_key: "ba6e9035c2f7617a5ba2252527be358f"
server_key_iv: "d58490eab966629d67e2a376194d581d"
server_key_version: "encryptionkey@0"
I20260928 02:57:42.049832 17400 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 02:57:42.053637 17415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:42.055334 17400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:42.055539 17400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "08f2ff0521d143698ce143139fdf202f"
format_stamp: "Formatted at 2026-09-28 02:57:42 on dist-test-slave-s1kg"
server_key: "ba6e9035c2f7617a5ba2252527be358f"
server_key_iv: "d58490eab966629d67e2a376194d581d"
server_key_version: "encryptionkey@0"
I20260928 02:57:42.055753 17400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:42.061774 17400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:42.066475 17400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:42.066802 17400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:42.085881 17400 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40101
I20260928 02:57:42.085937 17466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40101 every 8 connection(s)
I20260928 02:57:42.087316 17400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:42.090377 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17400
I20260928 02:57:42.090572 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:42.091141 28567 external_mini_cluster.cc:1428] Setting key 9044ba1fe8dd4b5071880f0f0d941fa5
I20260928 02:57:42.091583 17467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:42.099509 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:42.092948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:46327 (local address 127.0.0.1:40101)
0928 02:57:42.093753 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:42.093770 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:42.093821 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:57:42.095547 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:42.095580 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:42.096043 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:42.096513 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:42.097498 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:42.097506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:42.097523 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:42.097542 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:42.097636 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:42.097660 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:42.097879 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:42.098015 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:57:42.098216 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":320,"threads_started":1}
I20260928 02:57:42.103552 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:42.107832 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f: Bootstrap starting.
I20260928 02:57:42.112237 17467 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:42.112569 17470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:42.109415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:48165 (local address 127.0.0.1:40101)
0928 02:57:42.109658 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:42.109667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:42.110220 (+ 553us) server_negotiation.cc:407] Connection header received
0928 02:57:42.110391 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:42.110395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:42.110446 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:42.110576 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:42.111636 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:42.111640 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:42.111643 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:42.111660 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:42.111699 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:42.111713 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:42.111820 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:42.112158 (+ 338us) server_negotiation.cc:299] Negotiation successful
0928 02:57:42.112243 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 02:57:42.113700 17467 log.cc:824] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:42.116639 17467 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f: No bootstrap required, opened a new log
I20260928 02:57:42.124162 17467 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260928 02:57:42.124572 17467 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:42.124639 17467 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08f2ff0521d143698ce143139fdf202f, State: Initialized, Role: FOLLOWER
I20260928 02:57:42.125228 17467 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260928 02:57:42.125427 17467 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:42.125542 17467 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:42.125679 17467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:42.128331 17467 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260928 02:57:42.128816 17467 leader_election.cc:302] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08f2ff0521d143698ce143139fdf202f; no voters:
I20260928 02:57:42.129372 17467 leader_election.cc:288] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:42.129537 17477 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:42.130225 17477 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [term 1 LEADER]: Becoming Leader. State: Replica: 08f2ff0521d143698ce143139fdf202f, State: Running, Role: LEADER
I20260928 02:57:42.130759 17477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } }
I20260928 02:57:42.131608 17467 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:42.133605 17478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08f2ff0521d143698ce143139fdf202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260928 02:57:42.134011 17478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:42.134995 17479 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08f2ff0521d143698ce143139fdf202f. Latest consensus state: current_term: 1 leader_uuid: "08f2ff0521d143698ce143139fdf202f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08f2ff0521d143698ce143139fdf202f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40101 } } }
I20260928 02:57:42.135532 17479 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:42.136442 17482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:42.142309 17482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:42.152069 17482 catalog_manager.cc:1347] Generated new cluster ID: f050dab00b1f4333a5959aef3f9fce0c
I20260928 02:57:42.152206 17482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:42.170753 17482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:42.171252 17482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:42.186721 17482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08f2ff0521d143698ce143139fdf202f: Generated new TSK 0
I20260928 02:57:42.187654 17482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:42.196221 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17400
W20260928 02:57:42.197480 17462 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:46327
W20260928 02:57:42.198186 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17400: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:42.198338 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17400
2026-09-28T02:57:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:42 dist-test-slave-s1kg krb5kdc[17498](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:42 dist-test-slave-s1kg krb5kdc[17498](info): set up 2 sockets
Sep 28 02:57:42 dist-test-slave-s1kg krb5kdc[17498](info): commencing operation
krb5kdc: starting...
W20260928 02:57:43.319240 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:43 dist-test-slave-s1kg krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:43.591158 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35807
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:43.924239 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:43.924582 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:43.924751 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:43.932215 17511 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:43.932343 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:43.932468 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:43.932598 17511 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:43.932678 17511 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:43.943060 17511 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35807
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:43.944489 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:43.945847 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:43.958381 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:43.958433 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:43.958921 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:43.959394 17511 server_base.cc:1034] running on GCE node
I20260928 02:57:43.960289 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:43.961927 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:43.963157 17511 hybrid_clock.cc:648] HybridClock initialized: now 1790564263963099 us; error 69 us; skew 500 ppm
I20260928 02:57:43.963529 17511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:43 dist-test-slave-s1kg krb5kdc[17498](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:43.967298 17511 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:43.969048 17511 webserver.cc:460] Webserver started at http://127.0.0.1:41301/ using document root <none> and password file <none>
I20260928 02:57:43.969834 17511 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:43.969956 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:43.970310 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:43.973762 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "6676860da9ff4fdcbc701f465ccede4a"
format_stamp: "Formatted at 2026-09-28 02:57:43 on dist-test-slave-s1kg"
server_key: "c7b825ce47aaaa5cad3f3474879f5fb8"
server_key_iv: "970b02edbef46b5008e11085b07da2a6"
server_key_version: "encryptionkey@0"
I20260928 02:57:43.974661 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "6676860da9ff4fdcbc701f465ccede4a"
format_stamp: "Formatted at 2026-09-28 02:57:43 on dist-test-slave-s1kg"
server_key: "c7b825ce47aaaa5cad3f3474879f5fb8"
server_key_iv: "970b02edbef46b5008e11085b07da2a6"
server_key_version: "encryptionkey@0"
I20260928 02:57:43.980417 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:57:43.984494 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:43.986049 17511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:43.986248 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "6676860da9ff4fdcbc701f465ccede4a"
format_stamp: "Formatted at 2026-09-28 02:57:43 on dist-test-slave-s1kg"
server_key: "c7b825ce47aaaa5cad3f3474879f5fb8"
server_key_iv: "970b02edbef46b5008e11085b07da2a6"
server_key_version: "encryptionkey@0"
I20260928 02:57:43.986457 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:44.002154 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:44.006377 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:44.006681 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:44.026464 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35807
I20260928 02:57:44.026505 17577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35807 every 8 connection(s)
I20260928 02:57:44.027963 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:44.031713 17578 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:44.037174 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17511
I20260928 02:57:44.037564 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:44.037997 28567 external_mini_cluster.cc:1428] Setting key ed920fe46d80807687151e5eadb57592
I20260928 02:57:44.044023 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a: Bootstrap starting.
I20260928 02:57:44.048069 17578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:44.049165 17578 log.cc:824] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:44.052345 17578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a: No bootstrap required, opened a new log
Sep 28 02:57:44 dist-test-slave-s1kg krb5kdc[17498](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 02:57:44.060715 17578 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260928 02:57:44.061161 17578 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:44.061299 17578 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6676860da9ff4fdcbc701f465ccede4a, State: Initialized, Role: FOLLOWER
I20260928 02:57:44.061990 17578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260928 02:57:44.062268 17578 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:44.062425 17578 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:44.062606 17578 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:44.063459 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:44.040047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:40901 (local address 127.0.0.1:35807)
0928 02:57:44.040945 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:44.040962 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:44.041016 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:57:44.043477 (+ 2461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:44.043506 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:44.044052 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:44.044530 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:44.045948 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:44.047978 (+ 2030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:44.049476 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:44.050402 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:44.050620 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:44.054999 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:44.055012 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:44.055029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:44.055067 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:44.058504 (+ 3437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:44.059502 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:44.059509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:44.059515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:44.059696 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:44.060207 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:44.060210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:44.060212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:44.060776 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:44.061104 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:44.061647 (+ 543us) server_negotiation.cc:299] Negotiation successful
0928 02:57:44.062159 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":385,"threads_started":1}
I20260928 02:57:44.065587 17578 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260928 02:57:44.066143 17578 leader_election.cc:302] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6676860da9ff4fdcbc701f465ccede4a; no voters:
I20260928 02:57:44.066876 17578 leader_election.cc:288] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:44.067117 17583 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:44.067917 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:44.069231 17578 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:44.069881 17583 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [term 1 LEADER]: Becoming Leader. State: Replica: 6676860da9ff4fdcbc701f465ccede4a, State: Running, Role: LEADER
I20260928 02:57:44.070547 17583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } }
I20260928 02:57:44.081071 17585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6676860da9ff4fdcbc701f465ccede4a. Latest consensus state: current_term: 1 leader_uuid: "6676860da9ff4fdcbc701f465ccede4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } } }
I20260928 02:57:44.081535 17585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:44.094436 17584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6676860da9ff4fdcbc701f465ccede4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6676860da9ff4fdcbc701f465ccede4a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35807 } } }
I20260928 02:57:44.094811 17584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:44.095494 17599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:44.100371 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:44.073759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:37003 (local address 127.0.0.1:35807)
0928 02:57:44.073999 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:44.074008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:44.075553 (+ 1545us) server_negotiation.cc:407] Connection header received
0928 02:57:44.075805 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:44.075813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:44.075928 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:44.076093 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:44.078678 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:44.079941 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:44.082355 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:44.083365 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:44.083658 (+ 293us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:44.088383 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:44.088396 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:44.088404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:44.088466 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:44.094726 (+ 6260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:44.095996 (+ 1270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:44.096002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:44.096005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:44.096179 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:44.096851 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:44.096860 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:44.096867 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:44.097456 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:44.099207 (+ 1751us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:44.099494 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 02:57:44.099785 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:57:44.101292 17599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:44.111634 17599 catalog_manager.cc:1347] Generated new cluster ID: 21c52f0462d24d5c810b46b3ea3e172d
I20260928 02:57:44.111774 17599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:44.131079 17599 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:44.132507 17599 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:44.141080 17599 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6676860da9ff4fdcbc701f465ccede4a: Generated new TSK 0
I20260928 02:57:44.141973 17599 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:44.153905 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17511
I20260928 02:57:44.155166 17573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:44.247254 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17511
W20260928 02:57:44.265235 17496 connection.cc:535] client connection to 127.0.0.1:35807 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35807): Connection reset by peer (error 104)
2026-09-28T02:57:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:44 dist-test-slave-s1kg krb5kdc[17610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:44 dist-test-slave-s1kg krb5kdc[17610](info): set up 2 sockets
Sep 28 02:57:44 dist-test-slave-s1kg krb5kdc[17610](info): commencing operation
krb5kdc: starting...
W20260928 02:57:45.365294 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:45 dist-test-slave-s1kg krb5kdc[17610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:45.643652 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:45.992313 17623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:45.992619 17623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:45.992815 17623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:46.000084 17623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:46.000195 17623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:46.000294 17623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:46.000412 17623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:46.000559 17623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:46.010633 17623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42717
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:46.011965 17623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:46.013192 17623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:46.025997 17628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:46.026084 17629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:46.026084 17631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:46.026929 17623 server_base.cc:1034] running on GCE node
I20260928 02:57:46.027803 17623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:46.029337 17623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:46.030596 17623 hybrid_clock.cc:648] HybridClock initialized: now 1790564266030536 us; error 65 us; skew 500 ppm
I20260928 02:57:46.031041 17623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:46 dist-test-slave-s1kg krb5kdc[17610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:46.037748 17623 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:46.039458 17623 webserver.cc:460] Webserver started at http://127.0.0.1:36737/ using document root <none> and password file <none>
I20260928 02:57:46.040159 17623 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:46.040307 17623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:46.040620 17623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:46.043795 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "1d7b41577fc4468da58090f7abcf8346"
format_stamp: "Formatted at 2026-09-28 02:57:46 on dist-test-slave-s1kg"
server_key: "9a741c882f206c8de287bfe2fdec8b3d"
server_key_iv: "c5ef049eee0efcaf3b88e17a87cfe78f"
server_key_version: "encryptionkey@0"
I20260928 02:57:46.044605 17623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "1d7b41577fc4468da58090f7abcf8346"
format_stamp: "Formatted at 2026-09-28 02:57:46 on dist-test-slave-s1kg"
server_key: "9a741c882f206c8de287bfe2fdec8b3d"
server_key_iv: "c5ef049eee0efcaf3b88e17a87cfe78f"
server_key_version: "encryptionkey@0"
I20260928 02:57:46.050037 17623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:57:46.054078 17638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:46.055615 17623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:46.055811 17623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "1d7b41577fc4468da58090f7abcf8346"
format_stamp: "Formatted at 2026-09-28 02:57:46 on dist-test-slave-s1kg"
server_key: "9a741c882f206c8de287bfe2fdec8b3d"
server_key_iv: "c5ef049eee0efcaf3b88e17a87cfe78f"
server_key_version: "encryptionkey@0"
I20260928 02:57:46.056023 17623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:46.071974 17623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:46.076181 17623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:46.076470 17623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:46.095709 17623 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42717
I20260928 02:57:46.095759 17689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42717 every 8 connection(s)
I20260928 02:57:46.097106 17623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:46.100409 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17623
I20260928 02:57:46.100607 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:46.101133 17690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:46.101140 28567 external_mini_cluster.cc:1428] Setting key b05e36a2050a46a7c8ad95c8d7c6a117
I20260928 02:57:46.115195 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:46.103087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:56493 (local address 127.0.0.1:42717)
0928 02:57:46.103884 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:46.103894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:46.103936 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:57:46.105830 (+ 1894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:46.105851 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:46.106250 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:46.106574 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:46.107262 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:46.108638 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:46.110478 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:46.111213 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:46.111340 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:46.112748 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:46.112756 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:46.112775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:46.113365 (+ 590us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:46.113452 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:46.113477 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:46.113623 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:46.113924 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 02:57:46.114121 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":351,"threads_started":1}
I20260928 02:57:46.115556 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346: Bootstrap starting.
I20260928 02:57:46.118788 17690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:46.119065 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:46.119884 17690 log.cc:824] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:46.122992 17690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346: No bootstrap required, opened a new log
I20260928 02:57:46.130690 17693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:46.123880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:41147 (local address 127.0.0.1:42717)
0928 02:57:46.124077 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:46.124083 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:46.124505 (+ 422us) server_negotiation.cc:407] Connection header received
0928 02:57:46.124659 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:46.124665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:46.124759 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:46.124873 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:46.125343 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:46.126566 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:46.127899 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:46.128624 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:46.128748 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:46.129842 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:46.129849 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:46.129854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:46.129876 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:46.129930 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:46.129950 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:46.130102 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:46.130286 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 02:57:46.130339 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:57:46.133030 17690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } }
I20260928 02:57:46.133383 17690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:46.133450 17690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d7b41577fc4468da58090f7abcf8346, State: Initialized, Role: FOLLOWER
I20260928 02:57:46.134086 17690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } }
I20260928 02:57:46.134385 17690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:46.134552 17690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:46.134750 17690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:46.136956 17690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } }
I20260928 02:57:46.137470 17690 leader_election.cc:302] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d7b41577fc4468da58090f7abcf8346; no voters:
I20260928 02:57:46.138127 17690 leader_election.cc:288] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:46.138262 17700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:46.139042 17700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [term 1 LEADER]: Becoming Leader. State: Replica: 1d7b41577fc4468da58090f7abcf8346, State: Running, Role: LEADER
I20260928 02:57:46.140103 17700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } }
I20260928 02:57:46.140456 17690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:46.142892 17702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d7b41577fc4468da58090f7abcf8346. Latest consensus state: current_term: 1 leader_uuid: "1d7b41577fc4468da58090f7abcf8346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } } }
I20260928 02:57:46.142921 17701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d7b41577fc4468da58090f7abcf8346" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d7b41577fc4468da58090f7abcf8346" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42717 } } }
I20260928 02:57:46.143349 17702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:46.143350 17701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:46.144272 17705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:46.148988 17705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:46.158183 17705 catalog_manager.cc:1347] Generated new cluster ID: 2e1058f2006f489287e869ca12285386
I20260928 02:57:46.158300 17705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:46.190285 17705 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:46.192246 17705 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:46.214490 17705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d7b41577fc4468da58090f7abcf8346: Generated new TSK 0
I20260928 02:57:46.215780 17705 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:46.232302 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17623
W20260928 02:57:46.233548 17685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:56493
W20260928 02:57:46.234418 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17623: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:46.234584 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17623
W20260928 02:57:46.252137 17608 connection.cc:535] client connection to 127.0.0.1:42717 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42717): Connection reset by peer (error 104)
2026-09-28T02:57:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:46 dist-test-slave-s1kg krb5kdc[17721](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:46 dist-test-slave-s1kg krb5kdc[17721](info): set up 2 sockets
Sep 28 02:57:46 dist-test-slave-s1kg krb5kdc[17721](info): commencing operation
krb5kdc: starting...
W20260928 02:57:47.372954 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:47 dist-test-slave-s1kg krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 02:57:47.648197 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35201
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:47.993510 17734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:47.993844 17734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:47.994015 17734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:48.001368 17734 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:48.001542 17734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:48.001652 17734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:48.001747 17734 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:48.001840 17734 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:48.011981 17734 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35201
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:48.013350 17734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:48.014654 17734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:48.026913 17739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:48.026958 17740 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:48.027652 17742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:48.027755 17734 server_base.cc:1034] running on GCE node
I20260928 02:57:48.028596 17734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:48.030016 17734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:48.031220 17734 hybrid_clock.cc:648] HybridClock initialized: now 1790564268031168 us; error 48 us; skew 500 ppm
I20260928 02:57:48.031599 17734 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:48 dist-test-slave-s1kg krb5kdc[17721](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:48.035022 17734 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 02:57:48.036546 17734 webserver.cc:460] Webserver started at http://127.0.0.1:38777/ using document root <none> and password file <none>
I20260928 02:57:48.037266 17734 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:48.037413 17734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:48.037729 17734 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:48.040807 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "59a24ac42b98420385c4cbe3c17dd0fb"
format_stamp: "Formatted at 2026-09-28 02:57:48 on dist-test-slave-s1kg"
server_key: "2629fba4d99375e520a3958ecfb91c20"
server_key_iv: "defcf8b7d2dcc4af178d0bc7e872b147"
server_key_version: "encryptionkey@0"
I20260928 02:57:48.041602 17734 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "59a24ac42b98420385c4cbe3c17dd0fb"
format_stamp: "Formatted at 2026-09-28 02:57:48 on dist-test-slave-s1kg"
server_key: "2629fba4d99375e520a3958ecfb91c20"
server_key_iv: "defcf8b7d2dcc4af178d0bc7e872b147"
server_key_version: "encryptionkey@0"
I20260928 02:57:48.047145 17734 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260928 02:57:48.051085 17749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:48.052557 17734 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:48.052803 17734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "59a24ac42b98420385c4cbe3c17dd0fb"
format_stamp: "Formatted at 2026-09-28 02:57:48 on dist-test-slave-s1kg"
server_key: "2629fba4d99375e520a3958ecfb91c20"
server_key_iv: "defcf8b7d2dcc4af178d0bc7e872b147"
server_key_version: "encryptionkey@0"
I20260928 02:57:48.053032 17734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:48.059038 17734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:48.063760 17734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:48.064078 17734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:48.083477 17734 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35201
I20260928 02:57:48.083563 17800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35201 every 8 connection(s)
I20260928 02:57:48.084923 17734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:48.088689 17801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:48.094970 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17734
I20260928 02:57:48.095171 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:48.095592 28567 external_mini_cluster.cc:1428] Setting key 0c03d18ef3b95fcf0a89bfa4e593360a
I20260928 02:57:48.100606 17801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb: Bootstrap starting.
I20260928 02:57:48.104765 17801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:48.104966 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:48.097606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:46541 (local address 127.0.0.1:35201)
0928 02:57:48.098443 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:48.098458 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:48.098506 (+ 48us) server_negotiation.cc:407] Connection header received
0928 02:57:48.100961 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:48.100991 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:48.101471 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:48.101905 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:48.103006 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:48.103016 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:48.103034 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:48.103052 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:48.103130 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:48.103163 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:48.103371 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:48.103507 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 02:57:48.103707 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":328,"threads_started":1}
I20260928 02:57:48.105952 17801 log.cc:824] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:48.109424 17801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb: No bootstrap required, opened a new log
I20260928 02:57:48.109719 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:48.116936 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:48.114111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.197:40117 (local address 127.0.0.1:35201)
0928 02:57:48.114372 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:48.114380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:48.114756 (+ 376us) server_negotiation.cc:407] Connection header received
0928 02:57:48.114976 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:48.114981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:48.115036 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:48.115136 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:48.116171 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:48.116176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:48.116179 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:48.116195 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:48.116237 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:48.116253 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:48.116385 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:48.116508 (+ 123us) server_negotiation.cc:299] Negotiation successful
0928 02:57:48.116560 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:57:48.119092 17801 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260928 02:57:48.119511 17801 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:48.119632 17801 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59a24ac42b98420385c4cbe3c17dd0fb, State: Initialized, Role: FOLLOWER
I20260928 02:57:48.120283 17801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260928 02:57:48.120519 17801 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:48.120645 17801 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:48.120842 17801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:48.123558 17801 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260928 02:57:48.124097 17801 leader_election.cc:302] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59a24ac42b98420385c4cbe3c17dd0fb; no voters:
I20260928 02:57:48.124782 17801 leader_election.cc:288] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:48.124943 17811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:48.126216 17811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [term 1 LEADER]: Becoming Leader. State: Replica: 59a24ac42b98420385c4cbe3c17dd0fb, State: Running, Role: LEADER
I20260928 02:57:48.126794 17811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } }
I20260928 02:57:48.127240 17801 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:48.129853 17813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59a24ac42b98420385c4cbe3c17dd0fb. Latest consensus state: current_term: 1 leader_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } } }
I20260928 02:57:48.130226 17813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:48.129983 17812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59a24ac42b98420385c4cbe3c17dd0fb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35201 } } }
I20260928 02:57:48.130412 17812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:48.131301 17818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:48.138319 17818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:48.149883 17818 catalog_manager.cc:1347] Generated new cluster ID: f3692d7312ec40908c2d64599c814aa7
I20260928 02:57:48.150070 17818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:48.164072 17818 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:48.164562 17818 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:48.170486 17818 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59a24ac42b98420385c4cbe3c17dd0fb: Generated new TSK 0
I20260928 02:57:48.171497 17818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:48.176681 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17734
W20260928 02:57:48.178092 17796 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.197:46541
W20260928 02:57:48.179008 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17734: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:48.179150 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17734
2026-09-28T02:57:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:48 dist-test-slave-s1kg krb5kdc[17832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:48 dist-test-slave-s1kg krb5kdc[17832](info): set up 2 sockets
Sep 28 02:57:48 dist-test-slave-s1kg krb5kdc[17832](info): commencing operation
krb5kdc: starting...
W20260928 02:57:49.295897 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:49 dist-test-slave-s1kg krb5kdc[17832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:49.569952 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45603
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:38635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:45603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:49.929224 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:49.929642 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:49.929819 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:49.937407 17845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:49.937539 17845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:49.937659 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:49.937770 17845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:49.937834 17845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:49.948271 17845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:38635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:45603
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:49.949731 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:49.951438 17845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:49.965224 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:49.965204 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:49.965267 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:49.966092 17845 server_base.cc:1034] running on GCE node
I20260928 02:57:49.967109 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:49.968747 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:49.970027 17845 hybrid_clock.cc:648] HybridClock initialized: now 1790564269969980 us; error 51 us; skew 500 ppm
I20260928 02:57:49.970608 17845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:49 dist-test-slave-s1kg krb5kdc[17832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564269, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:49.974159 17845 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:49.976006 17845 webserver.cc:460] Webserver started at http://127.27.229.254:34497/ using document root <none> and password file <none>
I20260928 02:57:49.976783 17845 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:49.976943 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:49.977262 17845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:49.980564 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "d139d61955e44616b60e88104256f0ee"
format_stamp: "Formatted at 2026-09-28 02:57:49 on dist-test-slave-s1kg"
server_key: "8fb96f2c06bfb859bb9d21da1d13fd39"
server_key_iv: "9ffa22489e4f238bf242721f691e9aef"
server_key_version: "encryptionkey@0"
I20260928 02:57:49.981362 17845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "d139d61955e44616b60e88104256f0ee"
format_stamp: "Formatted at 2026-09-28 02:57:49 on dist-test-slave-s1kg"
server_key: "8fb96f2c06bfb859bb9d21da1d13fd39"
server_key_iv: "9ffa22489e4f238bf242721f691e9aef"
server_key_version: "encryptionkey@0"
I20260928 02:57:49.986896 17845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 02:57:49.990710 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:49.992372 17845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 02:57:49.992676 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "d139d61955e44616b60e88104256f0ee"
format_stamp: "Formatted at 2026-09-28 02:57:49 on dist-test-slave-s1kg"
server_key: "8fb96f2c06bfb859bb9d21da1d13fd39"
server_key_iv: "9ffa22489e4f238bf242721f691e9aef"
server_key_version: "encryptionkey@0"
I20260928 02:57:49.992892 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:50.029292 17845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:50.034004 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:50.034426 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:50.053195 17845 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:45603
I20260928 02:57:50.053229 17911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:45603 every 8 connection(s)
I20260928 02:57:50.054705 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:50.057392 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17845
I20260928 02:57:50.057626 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:50.058082 28567 external_mini_cluster.cc:1428] Setting key a59345062c95927391b70bf03739d713
I20260928 02:57:50.058784 17912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:57:50 dist-test-slave-s1kg krb5kdc[17832](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:50.073271 17912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee: Bootstrap starting.
I20260928 02:57:50.077559 17912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:50.078821 17912 log.cc:824] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:50.082515 17912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee: No bootstrap required, opened a new log
I20260928 02:57:50.083091 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:50.059886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46593 (local address 127.27.229.254:45603)
0928 02:57:50.060705 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:50.060720 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:50.060771 (+ 51us) server_negotiation.cc:407] Connection header received
0928 02:57:50.062549 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:50.062584 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:50.063153 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:50.063630 (+ 477us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:50.065027 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:50.067139 (+ 2112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:50.068527 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:50.069592 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:50.069782 (+ 190us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:50.073988 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:50.073998 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:50.074016 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:50.074061 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:50.078179 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:50.079060 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:50.079068 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:50.079077 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:50.079289 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:50.079887 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:50.079891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:50.079894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:50.080580 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:50.080907 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:50.081351 (+ 444us) server_negotiation.cc:299] Negotiation successful
0928 02:57:50.081710 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":331,"threads_started":1}
I20260928 02:57:50.087795 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:50.091578 17912 raft_consensus.cc:348] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } }
I20260928 02:57:50.092088 17912 raft_consensus.cc:374] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:50.092283 17912 raft_consensus.cc:729] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d139d61955e44616b60e88104256f0ee, State: Initialized, Role: FOLLOWER
I20260928 02:57:50.093101 17912 consensus_queue.cc:260] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } }
I20260928 02:57:50.093369 17912 raft_consensus.cc:388] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:50.093533 17912 raft_consensus.cc:482] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:50.093719 17912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:50.096155 17912 raft_consensus.cc:504] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } }
I20260928 02:57:50.096715 17912 leader_election.cc:302] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d139d61955e44616b60e88104256f0ee; no voters:
I20260928 02:57:50.097558 17912 leader_election.cc:288] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:50.097703 17922 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:50.099321 17922 raft_consensus.cc:686] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [term 1 LEADER]: Becoming Leader. State: Replica: d139d61955e44616b60e88104256f0ee, State: Running, Role: LEADER
I20260928 02:57:50.099892 17922 consensus_queue.cc:237] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } }
I20260928 02:57:50.100618 17912 sys_catalog.cc:564] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:50.104451 17923 sys_catalog.cc:455] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d139d61955e44616b60e88104256f0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } } }
I20260928 02:57:50.104986 17923 sys_catalog.cc:458] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:50.105516 17924 sys_catalog.cc:455] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader d139d61955e44616b60e88104256f0ee. Latest consensus state: current_term: 1 leader_uuid: "d139d61955e44616b60e88104256f0ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d139d61955e44616b60e88104256f0ee" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45603 } } }
I20260928 02:57:50.105804 17924 sys_catalog.cc:458] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:50.107947 17930 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:50.111816 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:50.092275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50071 (local address 127.27.229.254:45603)
0928 02:57:50.092544 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:50.092553 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:50.092902 (+ 349us) server_negotiation.cc:407] Connection header received
0928 02:57:50.093141 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:50.093146 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:50.093209 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:50.093313 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:50.094888 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:50.095684 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:50.097105 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:50.097677 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:50.097838 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:50.101104 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:50.101111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:50.101115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:50.101190 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:50.106631 (+ 5441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:50.108080 (+ 1449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:50.108092 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:50.108095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:50.108302 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:50.108936 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:50.108941 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:50.108944 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:50.109359 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:50.109692 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:50.110915 (+ 1223us) server_negotiation.cc:299] Negotiation successful
0928 02:57:50.111199 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:57:50.114754 17930 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:50.125768 17930 catalog_manager.cc:1347] Generated new cluster ID: e4f8e65f57d4468cab62347928451290
I20260928 02:57:50.125916 17930 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:50.168718 17930 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:50.170882 17930 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:50.180962 17930 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d139d61955e44616b60e88104256f0ee: Generated new TSK 0
I20260928 02:57:50.182114 17930 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:50.196980 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17845
I20260928 02:57:50.198367 17907 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:50.295260 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17845
W20260928 02:57:50.313380 17830 connection.cc:535] client connection to 127.27.229.254:45603 recv error: Network error: failed to read from TLS socket (remote: 127.27.229.254:45603): Connection reset by peer (error 104)
2026-09-28T02:57:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2116 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:50 dist-test-slave-s1kg krb5kdc[17944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:50 dist-test-slave-s1kg krb5kdc[17944](info): set up 2 sockets
Sep 28 02:57:50 dist-test-slave-s1kg krb5kdc[17944](info): commencing operation
krb5kdc: starting...
W20260928 02:57:51.419847 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:51 dist-test-slave-s1kg krb5kdc[17944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:51.695304 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44359
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:44789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:44359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:52.042640 17957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:52.043043 17957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:52.043239 17957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:52.050985 17957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:52.051119 17957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:52.051205 17957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:52.051290 17957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:52.051357 17957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:52.062242 17957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:44789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:44359
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:52.063740 17957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:52.065126 17957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:52.078734 17963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:52.078704 17962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:52.078917 17965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:52.080365 17957 server_base.cc:1034] running on GCE node
I20260928 02:57:52.081671 17957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:52.083418 17957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:52.084702 17957 hybrid_clock.cc:648] HybridClock initialized: now 1790564272084650 us; error 38 us; skew 500 ppm
I20260928 02:57:52.085203 17957 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:52 dist-test-slave-s1kg krb5kdc[17944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564272, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:52.092636 17957 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:52.094368 17957 webserver.cc:460] Webserver started at http://127.27.229.254:36941/ using document root <none> and password file <none>
I20260928 02:57:52.095227 17957 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:52.095381 17957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:52.095700 17957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:52.098944 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "8f0858ff2cd440bd85fd166226fa759d"
format_stamp: "Formatted at 2026-09-28 02:57:52 on dist-test-slave-s1kg"
server_key: "a0852a0890a68c8b869ff94046109623"
server_key_iv: "38ec8e977f4d8a8705d99887d26e9e88"
server_key_version: "encryptionkey@0"
I20260928 02:57:52.099833 17957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "8f0858ff2cd440bd85fd166226fa759d"
format_stamp: "Formatted at 2026-09-28 02:57:52 on dist-test-slave-s1kg"
server_key: "a0852a0890a68c8b869ff94046109623"
server_key_iv: "38ec8e977f4d8a8705d99887d26e9e88"
server_key_version: "encryptionkey@0"
I20260928 02:57:52.105640 17957 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:57:52.109859 17972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:52.111519 17957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260928 02:57:52.111739 17957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "8f0858ff2cd440bd85fd166226fa759d"
format_stamp: "Formatted at 2026-09-28 02:57:52 on dist-test-slave-s1kg"
server_key: "a0852a0890a68c8b869ff94046109623"
server_key_iv: "38ec8e977f4d8a8705d99887d26e9e88"
server_key_version: "encryptionkey@0"
I20260928 02:57:52.111967 17957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:52.146538 17957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:52.152122 17957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:52.152477 17957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:52.175045 17957 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:44359
I20260928 02:57:52.175103 18023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:44359 every 8 connection(s)
I20260928 02:57:52.176568 17957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:52.181005 18024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:52.184990 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 17957
I20260928 02:57:52.185521 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:52.186213 28567 external_mini_cluster.cc:1428] Setting key 8aaf0022ba8ca6a1acb5d36a6c3abc09
I20260928 02:57:52.197352 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: Bootstrap starting.
I20260928 02:57:52.202611 18024 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:52.203923 18024 log.cc:824] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:52.204941 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:52.188486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57097 (local address 127.27.229.254:44359)
0928 02:57:52.189382 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:52.189399 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:52.189456 (+ 57us) server_negotiation.cc:407] Connection header received
0928 02:57:52.191780 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:52.191824 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:52.192500 (+ 676us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:52.193051 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:52.193678 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:52.195992 (+ 2314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:52.198500 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:52.199685 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:52.199915 (+ 230us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:52.201319 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:52.201332 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:52.201355 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:52.202135 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:52.202242 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:52.202279 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:52.202452 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:52.203235 (+ 783us) server_negotiation.cc:299] Negotiation successful
0928 02:57:52.203425 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":386,"threads_started":1}
I20260928 02:57:52.207276 18024 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: No bootstrap required, opened a new log
I20260928 02:57:52.209751 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:52.216886 18024 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } }
I20260928 02:57:52.217373 18024 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:52.217500 18024 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0858ff2cd440bd85fd166226fa759d, State: Initialized, Role: FOLLOWER
I20260928 02:57:52.218091 18024 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } }
I20260928 02:57:52.218338 18024 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:52.218483 18024 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:52.218670 18024 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:52.221349 18024 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } }
I20260928 02:57:52.221599 18027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:52.214512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47535 (local address 127.27.229.254:44359)
0928 02:57:52.214791 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:52.214801 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:52.215330 (+ 529us) server_negotiation.cc:407] Connection header received
0928 02:57:52.215512 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:52.215516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:52.215586 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:52.215728 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:52.216252 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:52.217109 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:52.218900 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:52.219358 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:52.219505 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:52.220769 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:52.220775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:52.220778 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:52.220797 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:52.220852 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:52.220869 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:52.220994 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:52.221217 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 02:57:52.221277 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:57:52.222065 18024 leader_election.cc:302] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f0858ff2cd440bd85fd166226fa759d; no voters:
I20260928 02:57:52.222783 18024 leader_election.cc:288] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:52.222954 18034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:52.225790 18024 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:52.229859 18034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [term 1 LEADER]: Becoming Leader. State: Replica: 8f0858ff2cd440bd85fd166226fa759d, State: Running, Role: LEADER
I20260928 02:57:52.230582 18034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } }
I20260928 02:57:52.233866 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f0858ff2cd440bd85fd166226fa759d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } } }
I20260928 02:57:52.234323 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:52.234956 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f0858ff2cd440bd85fd166226fa759d. Latest consensus state: current_term: 1 leader_uuid: "8f0858ff2cd440bd85fd166226fa759d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f0858ff2cd440bd85fd166226fa759d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44359 } } }
I20260928 02:57:52.235366 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d [sys.catalog]: This master's current role is: LEADER
W20260928 02:57:52.244287 18048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:57:52.244472 18048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:57:52.244594 18049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:52.246068 18049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:52.254751 18049 catalog_manager.cc:1347] Generated new cluster ID: a5572bdab4044a619062a602365965b8
I20260928 02:57:52.254909 18049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:52.281656 18049 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:52.283124 18049 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:52.294287 18049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f0858ff2cd440bd85fd166226fa759d: Generated new TSK 0
I20260928 02:57:52.295290 18049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:52.306219 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17957
W20260928 02:57:52.307441 18019 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57097
W20260928 02:57:52.308331 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 17957: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:52.308467 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 17957
W20260928 02:57:52.323500 17942 connection.cc:535] client connection to 127.27.229.254:44359 recv error: Network error: failed to read from TLS socket (remote: 127.27.229.254:44359): Connection reset by peer (error 104)
2026-09-28T02:57:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2010 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:52 dist-test-slave-s1kg krb5kdc[18055](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:52 dist-test-slave-s1kg krb5kdc[18055](info): set up 2 sockets
Sep 28 02:57:52 dist-test-slave-s1kg krb5kdc[18055](info): commencing operation
krb5kdc: starting...
W20260928 02:57:53.447408 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:53 dist-test-slave-s1kg krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:53.726819 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45267
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:45267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:54.074484 18068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:54.074806 18068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:54.074999 18068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:54.082523 18068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:54.082657 18068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:54.082703 18068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:54.082809 18068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:54.082929 18068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:54.093331 18068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:45267
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:45267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:54.094766 18068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:54.096195 18068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:54.108629 18073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:54.108596 18076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:54.108678 18074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:54.109645 18068 server_base.cc:1034] running on GCE node
I20260928 02:57:54.110476 18068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:54.112125 18068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:54.113407 18068 hybrid_clock.cc:648] HybridClock initialized: now 1790564274113328 us; error 75 us; skew 500 ppm
I20260928 02:57:54.113859 18068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:54 dist-test-slave-s1kg krb5kdc[18055](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564274, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:54.117458 18068 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:54.119369 18068 webserver.cc:460] Webserver started at http://127.27.229.254:33131/ using document root <none> and password file <none>
I20260928 02:57:54.120105 18068 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:54.120258 18068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:54.120574 18068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:54.123812 18068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "f0b827ede0534ae7afa5d0b1c979934d"
format_stamp: "Formatted at 2026-09-28 02:57:54 on dist-test-slave-s1kg"
server_key: "c2b6bb3bca5948e35594f416a88ba224"
server_key_iv: "e8942ac3f032e3513d7fd549ec18b0e8"
server_key_version: "encryptionkey@0"
I20260928 02:57:54.124629 18068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "f0b827ede0534ae7afa5d0b1c979934d"
format_stamp: "Formatted at 2026-09-28 02:57:54 on dist-test-slave-s1kg"
server_key: "c2b6bb3bca5948e35594f416a88ba224"
server_key_iv: "e8942ac3f032e3513d7fd549ec18b0e8"
server_key_version: "encryptionkey@0"
I20260928 02:57:54.130318 18068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260928 02:57:54.134153 18083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:54.135756 18068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:57:54.135967 18068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "f0b827ede0534ae7afa5d0b1c979934d"
format_stamp: "Formatted at 2026-09-28 02:57:54 on dist-test-slave-s1kg"
server_key: "c2b6bb3bca5948e35594f416a88ba224"
server_key_iv: "e8942ac3f032e3513d7fd549ec18b0e8"
server_key_version: "encryptionkey@0"
I20260928 02:57:54.136197 18068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:54.142237 18068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:54.146804 18068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:54.147178 18068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:54.166716 18068 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:45267
I20260928 02:57:54.166806 18134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:45267 every 8 connection(s)
I20260928 02:57:54.168241 18068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:54.171931 18135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:54.174803 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18068
I20260928 02:57:54.175079 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:54.175518 28567 external_mini_cluster.cc:1428] Setting key e89c9111e07362c97fbede3c82a1880e
I20260928 02:57:54.183673 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:54.177447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48023 (local address 127.27.229.254:45267)
0928 02:57:54.178308 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:54.178319 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:54.178359 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:57:54.180104 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:54.180127 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:54.180487 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:54.180821 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:54.181974 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:54.181984 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:54.182024 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:54.182050 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:54.182126 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:54.182149 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:54.182335 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:54.182461 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 02:57:54.182692 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":379,"threads_started":1}
I20260928 02:57:54.184259 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d: Bootstrap starting.
I20260928 02:57:54.187788 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:54.188066 18135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:54.189067 18135 log.cc:824] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:54.192162 18135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d: No bootstrap required, opened a new log
I20260928 02:57:54.195386 18138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:54.192693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39007 (local address 127.27.229.254:45267)
0928 02:57:54.192924 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:54.192934 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:54.193305 (+ 371us) server_negotiation.cc:407] Connection header received
0928 02:57:54.193477 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:54.193481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:54.193536 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:54.193629 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:54.194654 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:54.194658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:54.194661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:54.194676 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:54.194720 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:54.194736 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:54.194892 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:54.194979 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 02:57:54.195055 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:57:54.203105 18135 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } }
I20260928 02:57:54.203624 18135 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:54.203745 18135 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0b827ede0534ae7afa5d0b1c979934d, State: Initialized, Role: FOLLOWER
I20260928 02:57:54.204497 18135 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } }
I20260928 02:57:54.204836 18135 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:54.205029 18135 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:54.205259 18135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:54.207950 18135 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } }
I20260928 02:57:54.208810 18135 leader_election.cc:302] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0b827ede0534ae7afa5d0b1c979934d; no voters:
I20260928 02:57:54.209664 18135 leader_election.cc:288] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:54.209838 18145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:54.210561 18145 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [term 1 LEADER]: Becoming Leader. State: Replica: f0b827ede0534ae7afa5d0b1c979934d, State: Running, Role: LEADER
I20260928 02:57:54.211161 18145 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } }
I20260928 02:57:54.211930 18135 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:54.213968 18146 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0b827ede0534ae7afa5d0b1c979934d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } } }
I20260928 02:57:54.214231 18147 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0b827ede0534ae7afa5d0b1c979934d. Latest consensus state: current_term: 1 leader_uuid: "f0b827ede0534ae7afa5d0b1c979934d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0b827ede0534ae7afa5d0b1c979934d" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 45267 } } }
I20260928 02:57:54.214430 18146 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:54.214493 18147 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:54.220634 18151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:54.225844 18151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:54.237582 18151 catalog_manager.cc:1347] Generated new cluster ID: 309413aad1c54af9a94169b934776125
I20260928 02:57:54.237725 18151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:54.262048 18151 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:54.262543 18151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:54.269964 18151 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0b827ede0534ae7afa5d0b1c979934d: Generated new TSK 0
I20260928 02:57:54.270900 18151 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:54.275518 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18068
W20260928 02:57:54.276618 18130 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48023
W20260928 02:57:54.277364 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18068: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:54.277519 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18068
2026-09-28T02:57:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1968 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:54 dist-test-slave-s1kg krb5kdc[18166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:54 dist-test-slave-s1kg krb5kdc[18166](info): set up 2 sockets
Sep 28 02:57:54 dist-test-slave-s1kg krb5kdc[18166](info): commencing operation
krb5kdc: starting...
W20260928 02:57:55.391381 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:55 dist-test-slave-s1kg krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:55.666451 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39153
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:56.014801 18179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:56.015280 18179 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:56.015437 18179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:56.023142 18179 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:56.023306 18179 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:56.023415 18179 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:56.023526 18179 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:56.023604 18179 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:56.034163 18179 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39153
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:56.035594 18179 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:56.036926 18179 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:56.049417 18184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:56.049453 18185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:56.049379 18187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:56.049935 18179 server_base.cc:1034] running on GCE node
I20260928 02:57:56.051226 18179 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:56.052798 18179 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:56.054035 18179 hybrid_clock.cc:648] HybridClock initialized: now 1790564276053989 us; error 45 us; skew 500 ppm
I20260928 02:57:56.054476 18179 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:56 dist-test-slave-s1kg krb5kdc[18166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564276, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:56.059196 18179 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:56.060905 18179 webserver.cc:460] Webserver started at http://127.27.229.254:40875/ using document root <none> and password file <none>
I20260928 02:57:56.061658 18179 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:56.061812 18179 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:56.062131 18179 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:56.065359 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "b493498f9ea54f939df6e8101b880bed"
format_stamp: "Formatted at 2026-09-28 02:57:56 on dist-test-slave-s1kg"
server_key: "9d0f43b33610dea38e74ef7bd515dcca"
server_key_iv: "3b916dc4793f8ef07fb309ebe84bf183"
server_key_version: "encryptionkey@0"
I20260928 02:57:56.066184 18179 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "b493498f9ea54f939df6e8101b880bed"
format_stamp: "Formatted at 2026-09-28 02:57:56 on dist-test-slave-s1kg"
server_key: "9d0f43b33610dea38e74ef7bd515dcca"
server_key_iv: "3b916dc4793f8ef07fb309ebe84bf183"
server_key_version: "encryptionkey@0"
I20260928 02:57:56.071746 18179 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 02:57:56.075598 18194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:56.077193 18179 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:56.077415 18179 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "b493498f9ea54f939df6e8101b880bed"
format_stamp: "Formatted at 2026-09-28 02:57:56 on dist-test-slave-s1kg"
server_key: "9d0f43b33610dea38e74ef7bd515dcca"
server_key_iv: "3b916dc4793f8ef07fb309ebe84bf183"
server_key_version: "encryptionkey@0"
I20260928 02:57:56.077641 18179 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:56.095144 18179 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:56.099548 18179 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:56.099891 18179 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:56.120666 18179 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:39153
I20260928 02:57:56.120685 18245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:39153 every 8 connection(s)
I20260928 02:57:56.122123 18179 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:56.123773 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18179
I20260928 02:57:56.123966 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:56.124428 28567 external_mini_cluster.cc:1428] Setting key b72569991c3af489a45ec551ff3ff6e0
I20260928 02:57:56.126734 18246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 02:57:56 dist-test-slave-s1kg krb5kdc[18166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:57:56.140877 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed: Bootstrap starting.
I20260928 02:57:56.144739 18246 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:56.145964 18246 log.cc:824] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:56.149606 18246 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed: No bootstrap required, opened a new log
I20260928 02:57:56.150529 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:56.126381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55639 (local address 127.27.229.254:39153)
0928 02:57:56.127241 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:56.127257 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:56.127306 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:57:56.129663 (+ 2357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:56.129693 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:56.130283 (+ 590us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:56.130755 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:56.132281 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:56.134362 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:56.135826 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:56.136925 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:56.137132 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:56.142126 (+ 4994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:56.142136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:56.142152 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:56.142200 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:56.145779 (+ 3579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:56.146804 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:56.146813 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:56.146823 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:56.147142 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:56.147641 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:56.147645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:56.147647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:56.148186 (+ 539us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:56.148412 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:56.148918 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 02:57:56.149339 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":381,"threads_started":1}
I20260928 02:57:56.155067 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:56.159781 18246 raft_consensus.cc:348] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } }
I20260928 02:57:56.160288 18246 raft_consensus.cc:374] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:56.160435 18246 raft_consensus.cc:729] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b493498f9ea54f939df6e8101b880bed, State: Initialized, Role: FOLLOWER
I20260928 02:57:56.161144 18246 consensus_queue.cc:260] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } }
I20260928 02:57:56.161418 18246 raft_consensus.cc:388] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:56.161672 18246 raft_consensus.cc:482] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:56.161937 18246 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:56.164351 18246 raft_consensus.cc:504] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } }
I20260928 02:57:56.164958 18246 leader_election.cc:302] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b493498f9ea54f939df6e8101b880bed; no voters:
I20260928 02:57:56.165697 18246 leader_election.cc:288] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:56.165829 18256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:56.168205 18246 sys_catalog.cc:564] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:56.170104 18256 raft_consensus.cc:686] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [term 1 LEADER]: Becoming Leader. State: Replica: b493498f9ea54f939df6e8101b880bed, State: Running, Role: LEADER
I20260928 02:57:56.170866 18256 consensus_queue.cc:237] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } }
I20260928 02:57:56.176599 18258 sys_catalog.cc:455] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader b493498f9ea54f939df6e8101b880bed. Latest consensus state: current_term: 1 leader_uuid: "b493498f9ea54f939df6e8101b880bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } } }
I20260928 02:57:56.177155 18258 sys_catalog.cc:458] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:56.176755 18257 sys_catalog.cc:455] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b493498f9ea54f939df6e8101b880bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b493498f9ea54f939df6e8101b880bed" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39153 } } }
I20260928 02:57:56.177901 18257 sys_catalog.cc:458] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:56.179216 18249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:56.160777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49355 (local address 127.27.229.254:39153)
0928 02:57:56.161032 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:56.161038 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:56.161277 (+ 239us) server_negotiation.cc:407] Connection header received
0928 02:57:56.161504 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:56.161510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:56.161700 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:56.161834 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:56.163445 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:56.164167 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:56.165514 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:56.166193 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:56.166338 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:56.169616 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:56.169626 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:56.169634 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:57:56.169696 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:56.174324 (+ 4628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:56.175343 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:56.175349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:56.175353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:56.175889 (+ 536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:57:56.176435 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:57:56.176442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:57:56.176448 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:57:56.176947 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:56.178388 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:56.178535 (+ 147us) server_negotiation.cc:299] Negotiation successful
0928 02:57:56.178774 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:57:56.191169 18266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:56.195649 18266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:56.205324 18266 catalog_manager.cc:1347] Generated new cluster ID: 571ddab067f14fb8a6ae12baf3f458ef
I20260928 02:57:56.205473 18266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:56.234407 18266 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:56.236235 18266 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:56.250257 18266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b493498f9ea54f939df6e8101b880bed: Generated new TSK 0
I20260928 02:57:56.251389 18266 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:56.264621 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18179
I20260928 02:57:56.265822 18241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:57:56.362629 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18179
2026-09-28T02:57:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2089 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:56 dist-test-slave-s1kg krb5kdc[18278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:56 dist-test-slave-s1kg krb5kdc[18278](info): set up 2 sockets
Sep 28 02:57:56 dist-test-slave-s1kg krb5kdc[18278](info): commencing operation
krb5kdc: starting...
W20260928 02:57:57.499366 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.099s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:57 dist-test-slave-s1kg krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:57.775624 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44571
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:45385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:44571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:57:58.120267 18291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:57:58.120653 18291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:57:58.120857 18291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:57:58.128432 18291 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:57:58.128574 18291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:57:58.128623 18291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:57:58.128675 18291 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:57:58.128712 18291 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:57:58.138932 18291 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:45385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:44571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:44571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:57:58.140324 18291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:57:58.141703 18291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:57:58.154366 18297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:58.154536 18296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:57:58.154490 18299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:57:58.155552 18291 server_base.cc:1034] running on GCE node
I20260928 02:57:58.156447 18291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:57:58.158006 18291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:57:58.159230 18291 hybrid_clock.cc:648] HybridClock initialized: now 1790564278159181 us; error 41 us; skew 500 ppm
I20260928 02:57:58.159597 18291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:57:58 dist-test-slave-s1kg krb5kdc[18278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564278, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:57:58.167887 18291 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:57:58.169749 18291 webserver.cc:460] Webserver started at http://127.27.229.254:43467/ using document root <none> and password file <none>
I20260928 02:57:58.170496 18291 fs_manager.cc:359] Metadata directory not provided
I20260928 02:57:58.170631 18291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:57:58.171018 18291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:57:58.174137 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "d685317c95c743f88639061ac887af01"
format_stamp: "Formatted at 2026-09-28 02:57:58 on dist-test-slave-s1kg"
server_key: "616c3bd9ed8c4eeb1e554bdbfec18770"
server_key_iv: "43a06492d5639a797a4840f52d05e01c"
server_key_version: "encryptionkey@0"
I20260928 02:57:58.175061 18291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "d685317c95c743f88639061ac887af01"
format_stamp: "Formatted at 2026-09-28 02:57:58 on dist-test-slave-s1kg"
server_key: "616c3bd9ed8c4eeb1e554bdbfec18770"
server_key_iv: "43a06492d5639a797a4840f52d05e01c"
server_key_version: "encryptionkey@0"
I20260928 02:57:58.180595 18291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:57:58.184325 18306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:57:58.185834 18291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:57:58.186019 18291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "d685317c95c743f88639061ac887af01"
format_stamp: "Formatted at 2026-09-28 02:57:58 on dist-test-slave-s1kg"
server_key: "616c3bd9ed8c4eeb1e554bdbfec18770"
server_key_iv: "43a06492d5639a797a4840f52d05e01c"
server_key_version: "encryptionkey@0"
I20260928 02:57:58.186224 18291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:57:58.202594 18291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:57:58.207048 18291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:57:58.207351 18291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:57:58.226940 18291 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:44571
I20260928 02:57:58.226929 18357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:44571 every 8 connection(s)
I20260928 02:57:58.228327 18291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:57:58.232129 18358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:57:58.233071 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18291
I20260928 02:57:58.233259 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:57:58.233673 28567 external_mini_cluster.cc:1428] Setting key 4b4611f3c7a664c1347f61f1d4ebad5a
I20260928 02:57:58.245167 18358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01: Bootstrap starting.
I20260928 02:57:58.247025 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:58.235545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50591 (local address 127.27.229.254:44571)
0928 02:57:58.236306 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:58.236317 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:58.236357 (+ 40us) server_negotiation.cc:407] Connection header received
0928 02:57:58.238553 (+ 2196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:58.238579 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:58.239050 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:58.239457 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:58.239970 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:58.241216 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:58.242532 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:58.243319 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:58.243519 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:58.244730 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:58.244739 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:58.244754 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:58.245287 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:58.245410 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:58.245436 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:58.245558 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:58.245757 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 02:57:58.246022 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":338,"threads_started":1}
I20260928 02:57:58.248777 18358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01: Neither blocks nor log segments found. Creating new log.
I20260928 02:57:58.249966 18358 log.cc:824] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01: Log is configured to *not* fsync() on all Append() calls
I20260928 02:57:58.251135 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:57:58.253068 18358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01: No bootstrap required, opened a new log
I20260928 02:57:58.262799 18361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:57:58.255781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55023 (local address 127.27.229.254:44571)
0928 02:57:58.255968 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 02:57:58.255975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:57:58.256355 (+ 380us) server_negotiation.cc:407] Connection header received
0928 02:57:58.256509 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:57:58.256515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:57:58.256599 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:57:58.256717 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:57:58.257227 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:58.258397 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:58.259854 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:57:58.260586 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:57:58.260737 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:57:58.261923 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:57:58.261929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:57:58.261933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:57:58.261956 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:57:58.262018 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:57:58.262041 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:57:58.262134 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 02:57:58.262308 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 02:57:58.262401 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:57:58.262491 18358 raft_consensus.cc:348] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } }
I20260928 02:57:58.263159 18358 raft_consensus.cc:374] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:57:58.263279 18358 raft_consensus.cc:729] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d685317c95c743f88639061ac887af01, State: Initialized, Role: FOLLOWER
I20260928 02:57:58.263978 18358 consensus_queue.cc:260] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } }
I20260928 02:57:58.264253 18358 raft_consensus.cc:388] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:57:58.264417 18358 raft_consensus.cc:482] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:57:58.264622 18358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:57:58.266906 18358 raft_consensus.cc:504] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } }
I20260928 02:57:58.267501 18358 leader_election.cc:302] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d685317c95c743f88639061ac887af01; no voters:
I20260928 02:57:58.268137 18358 leader_election.cc:288] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:57:58.268287 18368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:57:58.269148 18368 raft_consensus.cc:686] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [term 1 LEADER]: Becoming Leader. State: Replica: d685317c95c743f88639061ac887af01, State: Running, Role: LEADER
I20260928 02:57:58.269778 18368 consensus_queue.cc:237] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } }
I20260928 02:57:58.270366 18358 sys_catalog.cc:564] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:57:58.272763 18370 sys_catalog.cc:455] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d685317c95c743f88639061ac887af01. Latest consensus state: current_term: 1 leader_uuid: "d685317c95c743f88639061ac887af01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } } }
I20260928 02:57:58.272825 18369 sys_catalog.cc:455] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d685317c95c743f88639061ac887af01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d685317c95c743f88639061ac887af01" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 44571 } } }
I20260928 02:57:58.273192 18370 sys_catalog.cc:458] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:58.273223 18369 sys_catalog.cc:458] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01 [sys.catalog]: This master's current role is: LEADER
I20260928 02:57:58.275467 18376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:57:58.281929 18376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:57:58.292635 18376 catalog_manager.cc:1347] Generated new cluster ID: 28f0db9d11ef4f6babf8d204c20ee53f
I20260928 02:57:58.292778 18376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:57:58.305984 18376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:57:58.307469 18376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:57:58.319038 18376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d685317c95c743f88639061ac887af01: Generated new TSK 0
I20260928 02:57:58.319962 18376 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:57:58.329176 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18291
W20260928 02:57:58.330497 18353 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50591
W20260928 02:57:58.331271 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18291: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:57:58.331421 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18291
2026-09-28T02:57:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1967 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:57:58 dist-test-slave-s1kg krb5kdc[18389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:57:58 dist-test-slave-s1kg krb5kdc[18389](info): set up 2 sockets
Sep 28 02:57:58 dist-test-slave-s1kg krb5kdc[18389](info): commencing operation
krb5kdc: starting...
W20260928 02:57:59.463358 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:57:59 dist-test-slave-s1kg krb5kdc[18389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564279, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:57:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:57:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:57:59.738996 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37891
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:00.097276 18402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:00.097604 18402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:00.097702 18402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:00.105350 18402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:00.105497 18402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:00.105553 18402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:00.105610 18402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:00.105656 18402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:00.115926 18402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:00.117249 18402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:00.118568 18402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:00.131824 18407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:00.131795 18408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:00.131880 18410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:00.132714 18402 server_base.cc:1034] running on GCE node
I20260928 02:58:00.133636 18402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:00.135311 18402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:00.136601 18402 hybrid_clock.cc:648] HybridClock initialized: now 1790564280136540 us; error 62 us; skew 500 ppm
I20260928 02:58:00.137058 18402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:00 dist-test-slave-s1kg krb5kdc[18389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564280, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:00.140712 18402 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:58:00.142442 18402 webserver.cc:460] Webserver started at http://127.27.229.254:37455/ using document root <none> and password file <none>
I20260928 02:58:00.143266 18402 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:00.143419 18402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:00.143740 18402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:00.146991 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "a329fe941ec94bf989230b4b40d8aa03"
format_stamp: "Formatted at 2026-09-28 02:58:00 on dist-test-slave-s1kg"
server_key: "207aba3d47e3d6af00726cc5d1454db0"
server_key_iv: "b700e12d2ce3d44d348b9934d38761f0"
server_key_version: "encryptionkey@0"
I20260928 02:58:00.147786 18402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "a329fe941ec94bf989230b4b40d8aa03"
format_stamp: "Formatted at 2026-09-28 02:58:00 on dist-test-slave-s1kg"
server_key: "207aba3d47e3d6af00726cc5d1454db0"
server_key_iv: "b700e12d2ce3d44d348b9934d38761f0"
server_key_version: "encryptionkey@0"
I20260928 02:58:00.153236 18402 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260928 02:58:00.157370 18417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:00.158973 18402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:00.159202 18402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "a329fe941ec94bf989230b4b40d8aa03"
format_stamp: "Formatted at 2026-09-28 02:58:00 on dist-test-slave-s1kg"
server_key: "207aba3d47e3d6af00726cc5d1454db0"
server_key_iv: "b700e12d2ce3d44d348b9934d38761f0"
server_key_version: "encryptionkey@0"
I20260928 02:58:00.159438 18402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:00.165973 18402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:00.170542 18402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:00.170917 18402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:00.189205 18402 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:37891
I20260928 02:58:00.189229 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:37891 every 8 connection(s)
I20260928 02:58:00.190685 18402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:00.195039 18469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:00.196020 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18402
I20260928 02:58:00.196291 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:58:00.196749 28567 external_mini_cluster.cc:1428] Setting key 0a5090176dc9fc852a5846effb6f679a
I20260928 02:58:00.208137 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:00.198697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42213 (local address 127.27.229.254:37891)
0928 02:58:00.199708 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:00.199730 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:00.199784 (+ 54us) server_negotiation.cc:407] Connection header received
0928 02:58:00.202569 (+ 2785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:00.202605 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:00.203200 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:00.203730 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:00.205894 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:00.205904 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:00.205923 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:00.205945 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:00.206030 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:00.206062 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:00.206331 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:00.206476 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 02:58:00.206696 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":428,"threads_started":1}
I20260928 02:58:00.211589 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03: Bootstrap starting.
I20260928 02:58:00.212419 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:00.215590 18469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:00.217424 18469 log.cc:824] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:00.220963 18472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:00.218325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38907 (local address 127.27.229.254:37891)
0928 02:58:00.218564 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:00.218573 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:00.218968 (+ 395us) server_negotiation.cc:407] Connection header received
0928 02:58:00.219192 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:00.219197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:00.219249 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:00.219367 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:00.220308 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:00.220315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:00.220326 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 02:58:00.220343 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:00.220381 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 02:58:00.220396 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:00.220520 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:00.220621 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 02:58:00.220687 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 02:58:00.221324 18469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03: No bootstrap required, opened a new log
I20260928 02:58:00.230201 18469 raft_consensus.cc:348] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } }
I20260928 02:58:00.230633 18469 raft_consensus.cc:374] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 02:58:00.230764 18469 raft_consensus.cc:729] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a329fe941ec94bf989230b4b40d8aa03, State: Initialized, Role: FOLLOWER
I20260928 02:58:00.231494 18469 consensus_queue.cc:260] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } }
I20260928 02:58:00.231765 18469 raft_consensus.cc:388] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 02:58:00.231930 18469 raft_consensus.cc:482] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 02:58:00.232175 18469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:00.235040 18469 raft_consensus.cc:504] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } }
I20260928 02:58:00.235679 18469 leader_election.cc:302] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a329fe941ec94bf989230b4b40d8aa03; no voters:
I20260928 02:58:00.236327 18469 leader_election.cc:288] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 02:58:00.236461 18479 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:00.237761 18479 raft_consensus.cc:686] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [term 1 LEADER]: Becoming Leader. State: Replica: a329fe941ec94bf989230b4b40d8aa03, State: Running, Role: LEADER
I20260928 02:58:00.238265 18479 consensus_queue.cc:237] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } }
I20260928 02:58:00.238734 18469 sys_catalog.cc:564] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:00.241351 18480 sys_catalog.cc:455] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a329fe941ec94bf989230b4b40d8aa03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } } }
I20260928 02:58:00.241290 18481 sys_catalog.cc:455] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a329fe941ec94bf989230b4b40d8aa03. Latest consensus state: current_term: 1 leader_uuid: "a329fe941ec94bf989230b4b40d8aa03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a329fe941ec94bf989230b4b40d8aa03" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 37891 } } }
I20260928 02:58:00.241950 18480 sys_catalog.cc:458] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:00.241950 18481 sys_catalog.cc:458] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:00.243191 18488 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:00.254130 18488 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:00.263571 18488 catalog_manager.cc:1347] Generated new cluster ID: 5afd7216e760407a9595550147cde2de
I20260928 02:58:00.263731 18488 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:00.282732 18488 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:00.283322 18488 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:00.299158 18488 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a329fe941ec94bf989230b4b40d8aa03: Generated new TSK 0
I20260928 02:58:00.300375 18488 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:00.311385 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18402
W20260928 02:58:00.313179 18464 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42213
W20260928 02:58:00.314077 28567 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18402: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260928 02:58:00.314275 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18402
2026-09-28T02:58:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1983 ms)
[----------] 18 tests from AuthTokenIssuingTest (36283 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:00 dist-test-slave-s1kg krb5kdc[18500](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:00 dist-test-slave-s1kg krb5kdc[18500](info): set up 2 sockets
Sep 28 02:58:00 dist-test-slave-s1kg krb5kdc[18500](info): commencing operation
krb5kdc: starting...
W20260928 02:58:01.435501 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:58:01 dist-test-slave-s1kg krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:58:01.707675 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35689
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:02.043521 18513 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:02.043959 18513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:02.044085 18513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:02.044229 18513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:02.051959 18513 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:02.052054 18513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:02.052147 18513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:02.052219 18513 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:02.052306 18513 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:02.062584 18513 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:35689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:02.063921 18513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:02.065238 18513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:02.077222 18521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:02.077376 18518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:02.077345 18519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:02.078416 18513 server_base.cc:1034] running on GCE node
I20260928 02:58:02.079353 18513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:02.080837 18513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:02.082094 18513 hybrid_clock.cc:648] HybridClock initialized: now 1790564282082017 us; error 42 us; skew 500 ppm
I20260928 02:58:02.082464 18513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:02.085840 18513 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:58:02.087574 18513 webserver.cc:460] Webserver started at http://127.27.229.254:37907/ using document root <none> and password file <none>
I20260928 02:58:02.088311 18513 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:02.088445 18513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:02.088771 18513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:02.091874 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "370f301928f44a83b178a110d2fd9ce8"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "4cb7f2a3f8288e7075ce38b62f26f63a"
server_key_iv: "93ab1b38d9c8cf31ec9e723564990a55"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.092698 18513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "370f301928f44a83b178a110d2fd9ce8"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "4cb7f2a3f8288e7075ce38b62f26f63a"
server_key_iv: "93ab1b38d9c8cf31ec9e723564990a55"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.097954 18513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 02:58:02.101745 18528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:02.103277 18513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 02:58:02.103470 18513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "370f301928f44a83b178a110d2fd9ce8"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "4cb7f2a3f8288e7075ce38b62f26f63a"
server_key_iv: "93ab1b38d9c8cf31ec9e723564990a55"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.103716 18513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:02.122965 18513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:02.127643 18513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:02.127954 18513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:02.148834 18513 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:35689
I20260928 02:58:02.148840 18579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:35689 every 8 connection(s)
I20260928 02:58:02.150372 18513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:02.153998 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18513
I20260928 02:58:02.154336 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:58:02.155041 28567 external_mini_cluster.cc:1428] Setting key 669dd889d202a45a5fe4129c050cdc10
I20260928 02:58:02.155383 18580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:02.162057 18580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.253@KRBTEST.COM" created.
I20260928 02:58:02.184816 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.165511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35689 (local address 127.0.0.1:33840)
0928 02:58:02.166736 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.166776 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:02.168599 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:02.169428 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:02.169435 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:02.170152 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:02.170487 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.170504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.172090 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.172100 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.173141 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.173153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.174054 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.174060 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.174236 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.174999 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:02.175024 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:02.177154 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:02.180554 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.180564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.180575 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.180956 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.181380 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.181386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.181389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.181543 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.182325 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:02.182332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:02.182532 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 02:58:02.182781 (+ 249us) client_negotiation.cc:240] Negotiation successful
0928 02:58:02.183429 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":146,"spinlock_wait_cycles":16640,"thread_start_us":854,"threads_started":1}
I20260928 02:58:02.184840 18583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.165493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33840 (local address 127.27.229.254:35689)
0928 02:58:02.166301 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:02.166313 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:02.166932 (+ 619us) server_negotiation.cc:407] Connection header received
0928 02:58:02.168984 (+ 2052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:02.168990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:02.169210 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:02.169416 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:02.170700 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.171928 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.173283 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.173824 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.174018 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.177374 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:02.177382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:02.177393 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:02.177447 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:02.180419 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.181089 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.181104 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.181106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.181272 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.181659 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.181664 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.181673 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.182159 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:02.182385 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:02.182981 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 02:58:02.183283 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":388,"threads_started":1}
I20260928 02:58:02.187981 18580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:02.189059 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.188653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53752)
0928 02:58:02.188836 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.188941 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 02:58:02.189394 18532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
I20260928 02:58:02.190963 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.190550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53762)
0928 02:58:02.190716 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.190783 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:58:02.191249 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40793: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
I20260928 02:58:02.213840 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40793
--webserver_interface=127.27.229.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:58:02.240643 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } attempt: 1
I20260928 02:58:02.241945 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.241449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53766)
0928 02:58:02.241682 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.241808 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 02:58:02.243044 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.242666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53774)
0928 02:58:02.242799 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.242918 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 02:58:02.243301 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40793: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
I20260928 02:58:02.311561 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } attempt: 2
I20260928 02:58:02.312784 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.312250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53786)
0928 02:58:02.312514 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.312605 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:58:02.313778 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.313412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53788)
0928 02:58:02.313577 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.313647 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 02:58:02.313966 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40793: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
I20260928 02:58:02.405267 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } attempt: 3
I20260928 02:58:02.406534 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.406018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53790)
0928 02:58:02.406265 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.406352 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 02:58:02.407632 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.407203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53802)
0928 02:58:02.407387 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.407464 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260928 02:58:02.407824 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40793: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
I20260928 02:58:02.551090 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } attempt: 4
I20260928 02:58:02.552563 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.551966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53806)
0928 02:58:02.552296 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.552385 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 02:58:02.553575 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.553194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53822)
0928 02:58:02.553362 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.553448 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 02:58:02.553764 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40793: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40793: connect: Connection refused (error 111)
W20260928 02:58:02.556893 18587 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:02.557258 18587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:02.557380 18587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:02.557514 18587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:02.565299 18587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:02.565438 18587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:02.565569 18587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:02.565661 18587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:02.565771 18587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:02.576679 18587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.229.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:02.578045 18587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:02.579495 18587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:02.591773 18593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:02.591735 18595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:02.591817 18592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:02.596777 18587 server_base.cc:1034] running on GCE node
I20260928 02:58:02.597694 18587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:02.599195 18587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:02.600419 18587 hybrid_clock.cc:648] HybridClock initialized: now 1790564282600361 us; error 33 us; skew 500 ppm
I20260928 02:58:02.600720 18587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:02.604228 18587 init.cc:377] Logged in from keytab as kudu/127.27.229.253@KRBTEST.COM (short username kudu)
I20260928 02:58:02.605814 18587 webserver.cc:460] Webserver started at http://127.27.229.253:42993/ using document root <none> and password file <none>
I20260928 02:58:02.606551 18587 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:02.606640 18587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:02.607072 18587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:02.610116 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data/instance:
uuid: "a398ee208f364f77b1ebc65a33b7d574"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "3c3d52b2fef50406950ad2ba8e1044ab"
server_key_iv: "a8ac4cc7fb39dcff52f4bfeca263bc6e"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.611023 18587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal/instance:
uuid: "a398ee208f364f77b1ebc65a33b7d574"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "3c3d52b2fef50406950ad2ba8e1044ab"
server_key_iv: "a8ac4cc7fb39dcff52f4bfeca263bc6e"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.616349 18587 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260928 02:58:02.620108 18602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:02.621560 18587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 02:58:02.621801 18587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "a398ee208f364f77b1ebc65a33b7d574"
format_stamp: "Formatted at 2026-09-28 02:58:02 on dist-test-slave-s1kg"
server_key: "3c3d52b2fef50406950ad2ba8e1044ab"
server_key_iv: "a8ac4cc7fb39dcff52f4bfeca263bc6e"
server_key_version: "encryptionkey@0"
I20260928 02:58:02.622035 18587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:02.686323 18587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:02.690878 18587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:02.691129 18587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:02.710846 18587 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.253:40793
I20260928 02:58:02.710948 18653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.253:40793 every 8 connection(s)
I20260928 02:58:02.712257 18587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
I20260928 02:58:02.716252 18654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:02.721406 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18587
I20260928 02:58:02.721602 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-1/wal/instance
I20260928 02:58:02.722029 28567 external_mini_cluster.cc:1428] Setting key 16177898d4df2e2cbf20f890a43a6e81
I20260928 02:58:02.723860 18654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.252@KRBTEST.COM; defaulting to no policy
I20260928 02:58:02.749032 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.726651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33846 (local address 127.27.229.254:35689)
0928 02:58:02.727257 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:02.727265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:02.728357 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 02:58:02.730416 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:02.730424 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:02.730533 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:02.730675 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:02.732974 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.733787 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.736549 (+ 2762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.737016 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.737128 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.742247 (+ 5119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:02.742252 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:02.742255 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:02.742296 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:02.745194 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.746291 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.746296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.746298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.746431 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.746974 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.746978 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.746981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.747283 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:02.747399 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:02.748471 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0928 02:58:02.748717 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":293,"threads_started":1}
I20260928 02:58:02.750155 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.727323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35689 (local address 127.0.0.1:33846)
0928 02:58:02.728178 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.728222 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:02.730105 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:02.730977 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:02.730992 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:02.731819 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:02.732662 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.732689 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.733980 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.733989 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.736315 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.736336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.737300 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.737309 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.737833 (+ 524us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.738810 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:02.738878 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:02.741941 (+ 3063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:02.745369 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.745382 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.745405 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.746117 (+ 712us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.746563 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.746568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.746573 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.746824 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.747426 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:02.747437 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:02.747952 (+ 515us) client_negotiation.cc:769] Sending connection context
0928 02:58:02.748387 (+ 435us) client_negotiation.cc:240] Negotiation successful
0928 02:58:02.748953 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":376,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.252@KRBTEST.COM" created.
I20260928 02:58:02.752200 18654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:02.768939 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.752909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53830)
0928 02:58:02.753189 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.753208 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:02.753314 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:02.754405 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:02.754412 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:02.754942 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:02.755159 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.755173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.756982 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.756991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.758515 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.758533 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.759730 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.759738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.759968 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.760634 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:02.760652 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:02.763152 (+ 2500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:02.766438 (+ 3286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.766443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.766448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.766803 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.767209 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.767215 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.767219 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.767342 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.767951 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:02.767956 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:02.768090 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 02:58:02.768232 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:58:02.768521 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126}
I20260928 02:58:02.768998 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.753233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53830 (local address 127.27.229.253:40793)
0928 02:58:02.753922 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:02.753931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:02.753960 (+ 29us) server_negotiation.cc:407] Connection header received
0928 02:58:02.754010 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:02.754019 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:02.754234 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:02.754362 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:02.755346 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.756829 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.758679 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.759573 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.759696 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.763388 (+ 3692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:02.763401 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:02.763416 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:02.763444 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:02.766288 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.766967 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.766978 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.766981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.767100 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.767450 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.767455 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.767458 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.767807 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:02.768023 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:02.768393 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 02:58:02.768627 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":513,"thread_start_us":363,"threads_started":1}
I20260928 02:58:02.771477 18654 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:02.772676 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.772232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37136)
0928 02:58:02.772398 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.772524 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:58:02.772948 18604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.774528 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.774175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37148)
0928 02:58:02.774345 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.774429 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:58:02.774756 18654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
I20260928 02:58:02.786221 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:34371
--webserver_interface=127.27.229.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:40855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:58:02.824151 18654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 1
I20260928 02:58:02.825472 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.824879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37162)
0928 02:58:02.825115 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.825260 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:58:02.826646 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.826227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37176)
0928 02:58:02.826392 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.826500 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:58:02.826936 18654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.853019 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } attempt: 5
Sep 28 02:58:02 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:02.868505 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.853792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53832)
0928 02:58:02.854025 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.854053 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:02.854163 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:02.854631 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:02.854635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:02.855134 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:02.855318 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.855327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.856754 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.856758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.857689 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:02.857702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.858717 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.858722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:02.858972 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.859631 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:02.859648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:02.861638 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:02.865652 (+ 4014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.865657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.865660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.866070 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.866490 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:02.866494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:02.866534 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:02.866713 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:02.867730 (+ 1017us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:02.867735 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:02.867886 (+ 151us) client_negotiation.cc:769] Sending connection context
0928 02:58:02.868005 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 02:58:02.868248 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:58:02.869168 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.853952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.27.229.253:40793)
0928 02:58:02.854163 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:02.854173 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:02.854210 (+ 37us) server_negotiation.cc:407] Connection header received
0928 02:58:02.854294 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:02.854300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:02.854383 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:02.854503 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:02.855497 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.856581 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.857862 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:02.858542 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:02.858677 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:02.861851 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:02.861857 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:02.861861 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:02.861901 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:02.865484 (+ 3583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.866212 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.866218 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.866221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.866377 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:02.866896 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:02.866908 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:02.866911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:02.867446 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:02.867641 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:02.868211 (+ 570us) server_negotiation.cc:299] Negotiation successful
0928 02:58:02.868518 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:58:02.870169 18580 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:02.871381 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.870900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37182)
0928 02:58:02.871091 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.871191 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 02:58:02.872442 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.872010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37186)
0928 02:58:02.872184 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.872282 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 02:58:02.872676 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.895252 18654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 2
I20260928 02:58:02.896512 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.895957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37188)
0928 02:58:02.896192 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.896320 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260928 02:58:02.897593 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.897200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37202)
0928 02:58:02.897366 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.897452 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 02:58:02.897845 18654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.923928 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 1
I20260928 02:58:02.925313 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.924679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37216)
0928 02:58:02.924917 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.925053 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:58:02.926641 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.926240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37230)
0928 02:58:02.926416 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.926501 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:58:02.926944 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.989099 18654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 3
I20260928 02:58:02.990612 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.989924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37246)
0928 02:58:02.990224 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.990371 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 02:58:02.992175 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.991521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37252)
0928 02:58:02.991844 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.991965 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
W20260928 02:58:02.992480 18654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:02.995220 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 2
I20260928 02:58:02.996418 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.995868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37258)
0928 02:58:02.996106 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.996222 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 02:58:02.997502 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:02.997097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37264)
0928 02:58:02.997275 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:02.997360 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260928 02:58:02.997781 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
I20260928 02:58:03.104030 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 3
I20260928 02:58:03.105271 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.104706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37270)
0928 02:58:03.104940 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.105052 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:58:03.106559 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.106059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37280)
0928 02:58:03.106289 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.106384 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 02:58:03.106882 18580 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
W20260928 02:58:03.132829 18662 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:03.133203 18662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:03.133277 18662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:03.133359 18662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:58:03.135731 18654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 4
I20260928 02:58:03.136911 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.136388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37290)
0928 02:58:03.136612 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.136725 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 02:58:03.137979 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.137605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37294)
0928 02:58:03.137766 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.137849 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:58:03.138245 18654 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:34371: Network error: Client connection negotiation failed: client connection to 127.27.229.252:34371: connect: Connection refused (error 111)
W20260928 02:58:03.141610 18662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:03.141703 18662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:03.141824 18662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:03.141880 18662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:03.141963 18662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:03.152483 18662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:40855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:35689,127.27.229.253:40793,127.27.229.252:34371
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:34371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.229.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:03.153720 18662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:03.155094 18662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:03.167721 18667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:03.167845 18668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:03.168286 18670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:03.168524 18662 server_base.cc:1034] running on GCE node
I20260928 02:58:03.169402 18662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:03.171001 18662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:03.172282 18662 hybrid_clock.cc:648] HybridClock initialized: now 1790564283172210 us; error 47 us; skew 500 ppm
I20260928 02:58:03.172742 18662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:03.179667 18662 init.cc:377] Logged in from keytab as kudu/127.27.229.252@KRBTEST.COM (short username kudu)
I20260928 02:58:03.181474 18662 webserver.cc:460] Webserver started at http://127.27.229.252:33323/ using document root <none> and password file <none>
I20260928 02:58:03.182257 18662 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:03.182402 18662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:03.182745 18662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:03.185966 18662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data/instance:
uuid: "cca86950d1ca48adb22b9583be76c2ef"
format_stamp: "Formatted at 2026-09-28 02:58:03 on dist-test-slave-s1kg"
server_key: "ee8bf21b51b8b3bd8ea2aaad211f055d"
server_key_iv: "821aed091102e5fe2a57496128f8ab76"
server_key_version: "encryptionkey@0"
I20260928 02:58:03.186983 18662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal/instance:
uuid: "cca86950d1ca48adb22b9583be76c2ef"
format_stamp: "Formatted at 2026-09-28 02:58:03 on dist-test-slave-s1kg"
server_key: "ee8bf21b51b8b3bd8ea2aaad211f055d"
server_key_iv: "821aed091102e5fe2a57496128f8ab76"
server_key_version: "encryptionkey@0"
I20260928 02:58:03.192827 18662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 02:58:03.197367 18677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:03.199051 18662 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 02:58:03.199266 18662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "cca86950d1ca48adb22b9583be76c2ef"
format_stamp: "Formatted at 2026-09-28 02:58:03 on dist-test-slave-s1kg"
server_key: "ee8bf21b51b8b3bd8ea2aaad211f055d"
server_key_iv: "821aed091102e5fe2a57496128f8ab76"
server_key_version: "encryptionkey@0"
I20260928 02:58:03.199512 18662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:03.240298 18662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:03.245141 18662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:03.245460 18662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:03.265466 18662 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.252:34371
I20260928 02:58:03.265512 18728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.252:34371 every 8 connection(s)
I20260928 02:58:03.266944 18662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
I20260928 02:58:03.270709 18729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:03.273141 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18662
I20260928 02:58:03.273337 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790563915649057-28567-0/minicluster-data/master-2/wal/instance
I20260928 02:58:03.273846 28567 external_mini_cluster.cc:1428] Setting key c4a1d8317b929997a48880870b352f77
I20260928 02:58:03.279013 18729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:03.284121 18580 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 4
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:03.301184 18584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.284764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37308)
0928 02:58:03.285047 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.285067 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:03.285174 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:03.286022 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:03.286028 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:03.286612 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:03.286818 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.286854 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.288091 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.288097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.289520 (+ 1423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.289535 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.290714 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.290720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.290956 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.291555 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:03.291577 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:03.294113 (+ 2536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:03.298391 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.298401 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.298405 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.298943 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.299394 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.299400 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.299404 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.299565 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.300200 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:03.300204 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:03.300362 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:58:03.300504 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:58:03.300796 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260928 02:58:03.301741 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.275877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37306 (local address 127.27.229.252:34371)
0928 02:58:03.276724 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.276737 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.276786 (+ 49us) server_negotiation.cc:407] Connection header received
0928 02:58:03.278502 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.278525 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.279017 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.279360 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.281163 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.283026 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.284826 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.285879 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.286101 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.291744 (+ 5643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.291756 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.291770 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.291817 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.295911 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.296772 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.296782 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.296795 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.297073 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.297642 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.297652 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.297658 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.298552 (+ 894us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.298922 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.299628 (+ 706us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.300114 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":394,"threads_started":1}
I20260928 02:58:03.301748 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.284844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.27.229.252:34371)
0928 02:58:03.285703 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.285708 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.285731 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:58:03.285775 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.285781 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.285852 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.285956 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.287021 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.287950 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.289688 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.290539 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.290673 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.294356 (+ 3683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.294365 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.294370 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.294406 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.298238 (+ 3832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.299099 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.299104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.299107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.299258 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.299707 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.299713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.299718 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.300054 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.300210 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.300602 (+ 392us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.300796 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":727,"thread_start_us":526,"threads_started":1}
I20260928 02:58:03.302310 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.282043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.27.229.254:35689)
0928 02:58:03.282599 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.282606 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.283139 (+ 533us) server_negotiation.cc:407] Connection header received
0928 02:58:03.283345 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.283352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.283447 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.283582 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.285196 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.286312 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.288566 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.289237 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.289379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.294159 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.294169 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.294176 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.294231 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.297778 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.298692 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.298698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.298701 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.298905 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.299482 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.299488 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.299491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.300131 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.300271 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.301722 (+ 1451us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.301953 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":255,"threads_started":1}
I20260928 02:58:03.302527 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.282040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:35689 (local address 127.0.0.1:33854)
0928 02:58:03.282970 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.283014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:03.283207 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:03.283659 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:03.283664 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:03.284531 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:03.285014 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.285028 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.286479 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.286484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.288383 (+ 1899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.288401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.289502 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.289511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.289789 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.290518 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:03.290548 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:03.293921 (+ 3373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:03.297934 (+ 4013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.297946 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.297954 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.298516 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.299140 (+ 624us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.299147 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.299150 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.299329 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.300334 (+ 1005us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:03.300341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:03.300552 (+ 211us) client_negotiation.cc:769] Sending connection context
0928 02:58:03.300759 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 02:58:03.301095 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":33,"thread_start_us":509,"threads_started":1}
I20260928 02:58:03.304234 18729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:03.309553 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: Bootstrap starting.
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:03.313735 18580 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:03.314787 18580 log.cc:824] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: Log is configured to *not* fsync() on all Append() calls
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:58:03.318017 18580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: No bootstrap required, opened a new log
I20260928 02:58:03.319777 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.304936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40793 (local address 127.0.0.1:53844)
0928 02:58:03.305142 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.305163 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:03.305275 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:03.305990 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:03.305994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:03.306332 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:03.306472 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.306480 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.307751 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.307758 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.308732 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.308745 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.309454 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.309459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.309741 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.310346 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:03.310363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:03.312601 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:03.316881 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.316889 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.316893 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.317479 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.317889 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.317894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.317897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.318049 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.318741 (+ 692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:03.318746 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:03.318962 (+ 216us) client_negotiation.cc:769] Sending connection context
0928 02:58:03.319110 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 02:58:03.319409 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260928 02:58:03.319797 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.305369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.27.229.253:40793)
0928 02:58:03.305559 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.305563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.305578 (+ 15us) server_negotiation.cc:407] Connection header received
0928 02:58:03.305658 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.305663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.305743 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.305841 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.306629 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.307606 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.308880 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.309301 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.309442 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.312880 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.312889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.312894 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.312940 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.316714 (+ 3774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.317611 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.317616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.317620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.317767 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.318201 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.318213 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.318216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.318615 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.318782 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.319217 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.319425 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 02:58:03.320797 18729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:03.324611 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.305738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.27.229.254:35689)
0928 02:58:03.306761 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.306768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.306784 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:58:03.306868 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.306873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.306964 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.307061 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.308677 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.309741 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.311942 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.312401 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.312540 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.317558 (+ 5018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.317564 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.317569 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.317607 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.321393 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.322211 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.322222 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.322225 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.322373 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.322906 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.322913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.322919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.323328 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.323649 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.323991 (+ 342us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.324250 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876}
I20260928 02:58:03.329649 18580 raft_consensus.cc:348] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564283, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:03.330006 18580 raft_consensus.cc:729] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 370f301928f44a83b178a110d2fd9ce8, State: Initialized, Role: FOLLOWER
I20260928 02:58:03.330461 18580 consensus_queue.cc:260] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.331471 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.331951 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.332931 18580 sys_catalog.cc:564] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564281, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:03.337561 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.321612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37310)
0928 02:58:03.322111 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.322130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:03.322230 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:03.322500 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:03.322503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:03.322981 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:03.323173 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.323184 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.324562 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.324567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.325959 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.325974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.327128 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.327133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.327346 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.328022 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:03.328049 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:03.330698 (+ 2649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:03.334687 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.334694 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.334700 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.335254 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.335661 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.335667 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.335671 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.335803 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.336450 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:03.336454 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:03.336636 (+ 182us) client_negotiation.cc:769] Sending connection context
0928 02:58:03.336813 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 02:58:03.337141 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417}
I20260928 02:58:03.338579 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.321816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.27.229.252:34371)
0928 02:58:03.321998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.322004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.322315 (+ 311us) server_negotiation.cc:407] Connection header received
0928 02:58:03.322344 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.322346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.322389 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.322482 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.323331 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.324399 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.326100 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.326757 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.326941 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.330942 (+ 4001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.330949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.330953 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.330987 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.334527 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.335400 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.335409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.335415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.335568 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.335911 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.335918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.335921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.336252 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.336398 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.337943 (+ 1545us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.338205 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 02:58:03.343279 18729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: Bootstrap starting.
I20260928 02:58:03.346149 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.326333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.27.229.253:40793)
0928 02:58:03.326503 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.326509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.326561 (+ 52us) server_negotiation.cc:407] Connection header received
0928 02:58:03.326743 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.326748 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.326856 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.326955 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.328319 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.329040 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.330784 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.331514 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.331673 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.337475 (+ 5802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.337486 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.337491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.337537 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.342447 (+ 4910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.343435 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.343445 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.343451 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.343691 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.344163 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.344167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.344172 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.344636 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.344999 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.345324 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.345635 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:58:03.347433 18729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:03.348753 18729 log.cc:824] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:03.352167 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:03.354712 18729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: No bootstrap required, opened a new log
W20260928 02:58:03.361685 18748 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:58:03.361953 18748 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:58:03.364979 18729 raft_consensus.cc:348] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.365471 18729 raft_consensus.cc:729] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cca86950d1ca48adb22b9583be76c2ef, State: Initialized, Role: FOLLOWER
I20260928 02:58:03.366020 18729 consensus_queue.cc:260] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.370515 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.371026 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.371704 18555 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "370f301928f44a83b178a110d2fd9ce8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33862
I20260928 02:58:03.372066 18729 sys_catalog.cc:564] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:03.372159 18555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:58:03.372400 18555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:03.376281 18555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.378528 18555 leader_election.cc:288] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [CANDIDATE]: Term 1 election: Requested vote from peers a398ee208f364f77b1ebc65a33b7d574 (127.27.229.253:40793), cca86950d1ca48adb22b9583be76c2ef (127.27.229.252:34371)
I20260928 02:58:03.378906 18704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "370f301928f44a83b178a110d2fd9ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cca86950d1ca48adb22b9583be76c2ef"
I20260928 02:58:03.379405 18704 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [term 0 FOLLOWER]: Advancing to term 1
W20260928 02:58:03.381040 18534 tablet.cc:2369] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:03.382891 18704 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370f301928f44a83b178a110d2fd9ce8 in term 1.
I20260928 02:58:03.354031 18513 master_runner.cc:186] Error getting master registration for 127.27.229.253:40793: OK, instance_id { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" instance_seqno: 1790564282705008 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:58:03.383695 18513 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:58:03.383630 18530 leader_election.cc:302] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 370f301928f44a83b178a110d2fd9ce8, cca86950d1ca48adb22b9583be76c2ef; no voters:
I20260928 02:58:03.384199 18737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:03.391834 18737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [term 1 LEADER]: Becoming Leader. State: Replica: 370f301928f44a83b178a110d2fd9ce8, State: Running, Role: LEADER
I20260928 02:58:03.392720 18737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.398489 18756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370f301928f44a83b178a110d2fd9ce8. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.399187 18756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:03.407596 18761 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 02:58:03.410373 18761 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:03.437479 18654 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } attempt: 5
Sep 28 02:58:03 dist-test-slave-s1kg krb5kdc[18500](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564282, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:03.453065 18704 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [term 1 FOLLOWER]: Refusing update from remote peer 370f301928f44a83b178a110d2fd9ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:03.454546 18737 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:03.461709 18732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.438382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.27.229.252:34371)
0928 02:58:03.439020 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.439028 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.439052 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:03.439204 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.439210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.439291 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.439385 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.440518 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.441663 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.443490 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.444143 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.444270 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.449436 (+ 5166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.449445 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.449451 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.449501 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.453631 (+ 4130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.455610 (+ 1979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.455642 (+ 32us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.455645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.455918 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.456905 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.456911 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.456914 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.457871 (+ 957us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.458750 (+ 879us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.459221 (+ 471us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.460254 (+ 1033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260928 02:58:03.461802 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.438303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:34371 (local address 127.0.0.1:37322)
0928 02:58:03.438612 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:03.438635 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:03.438785 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:03.439573 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:03.439578 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:03.440110 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:03.440323 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.440336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.441840 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.441846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.443301 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:03.443317 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.444482 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.444487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:03.444788 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.445548 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:03.445571 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:03.449132 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:03.454538 (+ 5406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.454546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.454550 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.455408 (+ 858us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.456175 (+ 767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:03.456181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:03.456184 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:03.456501 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:03.458196 (+ 1695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:03.458201 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:03.458548 (+ 347us) client_negotiation.cc:769] Sending connection context
0928 02:58:03.459149 (+ 601us) client_negotiation.cc:240] Negotiation successful
0928 02:58:03.460320 (+ 1171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20260928 02:58:03.469339 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: Bootstrap starting.
I20260928 02:58:03.479355 18654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:03.482583 18654 log.cc:824] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: Log is configured to *not* fsync() on all Append() calls
W20260928 02:58:03.497839 18683 tablet.cc:2369] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:03.507968 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.403089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.27.229.252:34371)
0928 02:58:03.403286 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.403293 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.417200 (+ 13907us) server_negotiation.cc:407] Connection header received
0928 02:58:03.418896 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.418907 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.419028 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.419130 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.439645 (+ 20515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.441197 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.453838 (+ 12641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.454894 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.455022 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.479953 (+ 24931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.479964 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.479973 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.480043 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.486242 (+ 6199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.495285 (+ 9043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.495298 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.495304 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.495618 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.501416 (+ 5798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.501426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.501433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.501990 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.502411 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.506757 (+ 4346us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.507276 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 02:58:03.510057 18733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.409934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33876 (local address 127.27.229.254:35689)
0928 02:58:03.410208 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.410217 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.417414 (+ 7197us) server_negotiation.cc:407] Connection header received
0928 02:58:03.419529 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.419538 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.419651 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.419747 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.441539 (+ 21792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.443189 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.455458 (+ 12269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.456447 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.456578 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.484086 (+ 27508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.484097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.484106 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.484177 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.489611 (+ 5434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.495903 (+ 6292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.495914 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.495920 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.496148 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.500080 (+ 3932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.500090 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.500097 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.500699 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.501091 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.508970 (+ 7879us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.509434 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260928 02:58:03.529860 18654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: No bootstrap required, opened a new log
I20260928 02:58:03.535509 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.415798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.27.229.253:40793)
0928 02:58:03.416004 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.416010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.458574 (+ 42564us) server_negotiation.cc:407] Connection header received
0928 02:58:03.459855 (+ 1281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.459864 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.459972 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.460064 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.476068 (+ 16004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.477786 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.517949 (+ 40163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.519111 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.519276 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.525038 (+ 5762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:03.525047 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:03.525053 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:03.525105 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:03.529584 (+ 4479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.531406 (+ 1822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.531411 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.531414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.531597 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:03.532436 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:03.532442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:03.532445 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:03.532868 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:03.533140 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.534293 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.534780 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 02:58:03.528642 18662 master_runner.cc:186] Error getting master registration for 127.27.229.253:40793: OK, instance_id { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" instance_seqno: 1790564282705008 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:58:03.540337 18662 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:58:03.542717 18771 mvcc.cc:205] Tried to move back new op lower bound from 7334151304993144832 to 7334151304784531456. Current Snapshot: MvccSnapshot[applied={T|T < 7334151304993144832}]
I20260928 02:58:03.544149 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370f301928f44a83b178a110d2fd9ce8. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.544476 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.544505 18737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.544813 18737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:03.548687 18761 catalog_manager.cc:1347] Generated new cluster ID: 25468fa6cd8848cd87a91ec60bd73c22
I20260928 02:58:03.548806 18761 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:03.552028 18751 sys_catalog.cc:455] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.552318 18751 sys_catalog.cc:458] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef [sys.catalog]: This master's current role is: FOLLOWER
W20260928 02:58:03.561710 18773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:58:03.561858 18773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 02:58:03.564787 18654 raft_consensus.cc:348] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.565258 18654 raft_consensus.cc:729] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a398ee208f364f77b1ebc65a33b7d574, State: Initialized, Role: FOLLOWER
I20260928 02:58:03.565790 18654 consensus_queue.cc:260] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } }
I20260928 02:58:03.582386 18629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "370f301928f44a83b178a110d2fd9ce8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a398ee208f364f77b1ebc65a33b7d574"
I20260928 02:58:03.583017 18629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:03.582536 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.584502 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.586375 18629 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 370f301928f44a83b178a110d2fd9ce8 in term 1.
I20260928 02:58:03.582340 18654 sys_catalog.cc:564] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:03.587152 18628 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [term 1 FOLLOWER]: Refusing update from remote peer 370f301928f44a83b178a110d2fd9ce8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:03.589412 18756 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:03.621138 18781 sys_catalog.cc:455] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 370f301928f44a83b178a110d2fd9ce8. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.621610 18781 sys_catalog.cc:458] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.622069 18786 sys_catalog.cc:455] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.622653 18786 sys_catalog.cc:458] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:03.629405 18756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "370f301928f44a83b178a110d2fd9ce8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "370f301928f44a83b178a110d2fd9ce8" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 35689 } } peers { permanent_uuid: "a398ee208f364f77b1ebc65a33b7d574" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40793 } } peers { permanent_uuid: "cca86950d1ca48adb22b9583be76c2ef" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 34371 } } }
I20260928 02:58:03.629715 18756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:03.636175 18761 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:03.638789 18761 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:03.670037 18796 catalog_manager.cc:1259] Loaded cluster ID: 25468fa6cd8848cd87a91ec60bd73c22
I20260928 02:58:03.670176 18796 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: loading cluster ID for follower catalog manager: success
I20260928 02:58:03.675871 18761 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 370f301928f44a83b178a110d2fd9ce8: Generated new TSK 0
I20260928 02:58:03.677002 18761 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 02:58:03.677289 18796 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: acquiring CA information for follower catalog manager: success
I20260928 02:58:03.680233 18796 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a398ee208f364f77b1ebc65a33b7d574: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 02:58:03.816035 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:03.804053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.27.229.252:34371)
0928 02:58:03.804284 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.804291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.813865 (+ 9574us) server_negotiation.cc:407] Connection header received
0928 02:58:03.814369 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.814375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.814465 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.814541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.815735 (+ 1194us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37330: BlockingRecv error: recv got EOF from 127.0.0.1:37330 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 02:58:03.818662 18733 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:03.812135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.27.229.254:35689)
0928 02:58:03.812375 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.812383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.817070 (+ 4687us) server_negotiation.cc:407] Connection header received
0928 02:58:03.817324 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.817330 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.817411 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.817483 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.818389 (+ 906us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33888: BlockingRecv error: recv got EOF from 127.0.0.1:33888 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 02:58:03.822065 18659 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:03.819906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53872 (local address 127.27.229.253:40793)
0928 02:58:03.820134 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.820140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.820533 (+ 393us) server_negotiation.cc:407] Connection header received
0928 02:58:03.820793 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.820799 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.820876 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.820954 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.821819 (+ 865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53872: BlockingRecv error: recv got EOF from 127.0.0.1:53872 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 02:58:03.870319 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.854925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.27.229.253:40793)
0928 02:58:03.855129 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.855135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.862093 (+ 6958us) server_negotiation.cc:407] Connection header received
0928 02:58:03.862671 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.862677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.862937 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.863090 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:03.863834 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.865048 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.867153 (+ 2105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.867877 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.867975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.868771 (+ 796us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:03.869464 (+ 693us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:03.869603 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.869861 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.869966 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:58:03.916302 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:03.904924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37336 (local address 127.27.229.252:34371)
0928 02:58:03.905115 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.905122 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.914071 (+ 8949us) server_negotiation.cc:407] Connection header received
0928 02:58:03.914622 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.914629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.914764 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.914878 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:03.916030 (+ 1152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37336: BlockingRecv error: recv got EOF from 127.0.0.1:37336 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 02:58:03.965107 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:03.950436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.27.229.253:40793)
0928 02:58:03.950693 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:03.950700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:03.957831 (+ 7131us) server_negotiation.cc:407] Connection header received
0928 02:58:03.958382 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:03.958389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:03.958491 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:03.958558 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:03.959326 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.960546 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.962636 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:03.963392 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:03.963503 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:03.964166 (+ 663us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:03.964348 (+ 182us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:03.964426 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:03.964689 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 02:58:03.964785 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 02:58:04.025596 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.004571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37340 (local address 127.27.229.252:34371)
0928 02:58:04.004836 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.004846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.023544 (+ 18698us) server_negotiation.cc:407] Connection header received
0928 02:58:04.023813 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.023820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.023923 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.024025 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.025331 (+ 1306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37340: BlockingRecv error: recv got EOF from 127.0.0.1:37340 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 02:58:04.080004 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.064963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53892 (local address 127.27.229.253:40793)
0928 02:58:04.065164 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.065170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.072417 (+ 7247us) server_negotiation.cc:407] Connection header received
0928 02:58:04.072757 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.072764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.072858 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.072928 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.073730 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.074939 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.077552 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.078264 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.078376 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.079075 (+ 699us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.079244 (+ 169us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.079306 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.079532 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.079632 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:58:04.104995 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.098607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.27.229.252:34371)
0928 02:58:04.098821 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.098859 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.103259 (+ 4400us) server_negotiation.cc:407] Connection header received
0928 02:58:04.103470 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.103477 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.103597 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.103730 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.104722 (+ 992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 02:58:04.124137 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.116929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53906 (local address 127.27.229.253:40793)
0928 02:58:04.117128 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.117134 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.117700 (+ 566us) server_negotiation.cc:407] Connection header received
0928 02:58:04.117920 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.117926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.118018 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.118108 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.118611 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.119780 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.121822 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.122597 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.122748 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.123309 (+ 561us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.123465 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.123562 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.123721 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.123819 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:58:04.138957 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.131014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.27.229.252:34371)
0928 02:58:04.131301 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.131309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.137128 (+ 5819us) server_negotiation.cc:407] Connection header received
0928 02:58:04.137338 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.137345 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.137456 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.137582 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.138674 (+ 1092us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37372: BlockingRecv error: recv got EOF from 127.0.0.1:37372 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
I20260928 02:58:04.177414 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.170197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53922 (local address 127.27.229.253:40793)
0928 02:58:04.170450 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.170458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.170823 (+ 365us) server_negotiation.cc:407] Connection header received
0928 02:58:04.171064 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.171072 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.171181 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.171285 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.171771 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.172954 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.175118 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.175793 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.175912 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.176430 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.176637 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.176720 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.176931 (+ 211us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.177016 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 02:58:04.186340 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.184369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.27.229.252:34371)
0928 02:58:04.184562 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.184566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.185006 (+ 440us) server_negotiation.cc:407] Connection header received
0928 02:58:04.185155 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.185161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.185262 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.185378 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.186069 (+ 691us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:58:04.212798 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.206511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.27.229.253:40793)
0928 02:58:04.206736 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.206762 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.207088 (+ 326us) server_negotiation.cc:407] Connection header received
0928 02:58:04.207318 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.207322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.207419 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.207511 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.208041 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.208912 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.210912 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.211346 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.211449 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.212009 (+ 560us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.212148 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.212234 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.212420 (+ 186us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.212497 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260928 02:58:04.227859 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.225389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37400 (local address 127.27.229.252:34371)
0928 02:58:04.225732 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.225743 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.226387 (+ 644us) server_negotiation.cc:407] Connection header received
0928 02:58:04.226562 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.226570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.226682 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.226810 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.227627 (+ 817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37400: BlockingRecv error: recv got EOF from 127.0.0.1:37400 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:58:04.271046 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.263683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53932 (local address 127.27.229.253:40793)
0928 02:58:04.263940 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.263946 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.264281 (+ 335us) server_negotiation.cc:407] Connection header received
0928 02:58:04.264660 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.264666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.264755 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.264867 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.265470 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.266672 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.268797 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.269515 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.269662 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.270178 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.270330 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.270415 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.270602 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.270687 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260928 02:58:04.287413 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.285025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.27.229.252:34371)
0928 02:58:04.285229 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.285234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.285758 (+ 524us) server_negotiation.cc:407] Connection header received
0928 02:58:04.285973 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.285979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.286085 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.286202 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.287146 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37416: BlockingRecv error: recv got EOF from 127.0.0.1:37416 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 02:58:04.361542 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.355301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53948 (local address 127.27.229.253:40793)
0928 02:58:04.355524 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.355530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.355792 (+ 262us) server_negotiation.cc:407] Connection header received
0928 02:58:04.356031 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.356037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.356132 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.356236 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.356659 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.357419 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.358786 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.359462 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.359600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.360371 (+ 771us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.360519 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.360620 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.361092 (+ 472us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.361202 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 02:58:04.370820 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.368625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37428 (local address 127.27.229.252:34371)
0928 02:58:04.368862 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.368870 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.369175 (+ 305us) server_negotiation.cc:407] Connection header received
0928 02:58:04.369446 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.369451 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.369551 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.369660 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.370632 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37428: BlockingRecv error: recv got EOF from 127.0.0.1:37428 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 02:58:04.509151 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.503531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53960 (local address 127.27.229.253:40793)
0928 02:58:04.503852 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.503860 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.504251 (+ 391us) server_negotiation.cc:407] Connection header received
0928 02:58:04.504497 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.504502 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.504588 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.504714 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.505108 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.505840 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.507254 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.507755 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.507861 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.508343 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.508467 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.508539 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.508775 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.508845 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260928 02:58:04.517309 18735 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:04.515347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37436 (local address 127.27.229.252:34371)
0928 02:58:04.515592 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.515599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.515880 (+ 281us) server_negotiation.cc:407] Connection header received
0928 02:58:04.516099 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.516107 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.516194 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.516303 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:04.517083 (+ 780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37436: BlockingRecv error: recv got EOF from 127.0.0.1:37436 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 02:58:04.563819 18773 catalog_manager.cc:1259] Loaded cluster ID: 25468fa6cd8848cd87a91ec60bd73c22
I20260928 02:58:04.563922 18773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: loading cluster ID for follower catalog manager: success
I20260928 02:58:04.566658 18773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: acquiring CA information for follower catalog manager: success
I20260928 02:58:04.568596 18773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cca86950d1ca48adb22b9583be76c2ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:58:04.786017 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.780905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53968 (local address 127.27.229.253:40793)
0928 02:58:04.781164 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.781170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.781302 (+ 132us) server_negotiation.cc:407] Connection header received
0928 02:58:04.781551 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.781557 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.781632 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.781746 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.782188 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.782934 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.784323 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.784752 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.784888 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.785280 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.785408 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.785508 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.785670 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.785753 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 02:58:04.798291 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.792446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.27.229.252:34371)
0928 02:58:04.792673 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.792679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.792913 (+ 234us) server_negotiation.cc:407] Connection header received
0928 02:58:04.793236 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.793241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.793412 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.793515 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.793899 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.794608 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.796030 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.796449 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.796554 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.797105 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.797659 (+ 554us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.797774 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.797888 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.797968 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 02:58:04.812801 18735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.804855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37460 (local address 127.27.229.252:34371)
0928 02:58:04.805038 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.805044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.806207 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 02:58:04.806417 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.806422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.806506 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.806581 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.807343 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.808452 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.810725 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.811356 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.811493 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.812080 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.812182 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.812289 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.812399 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.812498 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 02:58:04.812834 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.805397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.27.229.253:40793)
0928 02:58:04.805586 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.805595 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.806477 (+ 882us) server_negotiation.cc:407] Connection header received
0928 02:58:04.806758 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.806765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.806881 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.806948 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.807758 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.808786 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.810435 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.811084 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.811221 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.811808 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.811916 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.812021 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.812224 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.812309 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 02:58:04.813544 18911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:04.804721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.27.229.254:35689)
0928 02:58:04.805766 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:04.805772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:04.805791 (+ 19us) server_negotiation.cc:407] Connection header received
0928 02:58:04.805911 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:04.805919 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:04.806192 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:04.806895 (+ 703us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:04.807216 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.808286 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.810766 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:04.811318 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:04.811493 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:04.812131 (+ 638us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:04.812688 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:04.812792 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:04.813136 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 02:58:04.813224 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":892,"thread_start_us":759,"threads_started":1}
I20260928 02:58:04.818923 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18513
I20260928 02:58:04.820266 18575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:04.933586 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18513
W20260928 02:58:04.954500 18678 connection.cc:535] server connection from 127.0.0.1:37308 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:58:04.954998 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18587
I20260928 02:58:04.956225 18649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:05.064481 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18587
I20260928 02:58:05.085327 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18662
I20260928 02:58:05.086483 18724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:05.195156 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18662
2026-09-28T02:58:05Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4884 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 02:58:05 dist-test-slave-s1kg krb5kdc[18919](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 02:58:05 dist-test-slave-s1kg krb5kdc[18919](info): set up 2 sockets
Sep 28 02:58:05 dist-test-slave-s1kg krb5kdc[18919](info): commencing operation
krb5kdc: starting...
W20260928 02:58:06.327502 28567 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/test-user.keytab.
Sep 28 02:58:06 dist-test-slave-s1kg krb5kdc[18919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T02:58:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T02:58:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.229.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
Entry for principal kudu/127.27.229.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab.
I20260928 02:58:06.602697 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39307
--webserver_interface=127.27.229.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 02:58:06.942361 18932 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:06.942723 18932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:06.942787 18932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:06.942939 18932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:06.951040 18932 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:06.951156 18932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:06.951244 18932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:06.951288 18932 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:06.951363 18932 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:06.962280 18932 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.254.keytab
--principal=kudu/127.27.229.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.254:39307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.229.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:06.963696 18932 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:06.965059 18932 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:06.977404 18937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:06.977490 18938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:06.977437 18940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:06.978849 18932 server_base.cc:1034] running on GCE node
I20260928 02:58:06.979809 18932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:06.981453 18932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:06.982798 18932 hybrid_clock.cc:648] HybridClock initialized: now 1790564286982735 us; error 43 us; skew 500 ppm
I20260928 02:58:06.983263 18932 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:06 dist-test-slave-s1kg krb5kdc[18919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:06.986904 18932 init.cc:377] Logged in from keytab as kudu/127.27.229.254@KRBTEST.COM (short username kudu)
I20260928 02:58:06.988504 18932 webserver.cc:460] Webserver started at http://127.27.229.254:33023/ using document root <none> and password file <none>
I20260928 02:58:06.989355 18932 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:06.989498 18932 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:06.989842 18932 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:06.993017 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data/instance:
uuid: "2a6db1624773436eaceb9f696d8b74b3"
format_stamp: "Formatted at 2026-09-28 02:58:06 on dist-test-slave-s1kg"
server_key: "42f89902d112015308efcdb9a47e1100"
server_key_iv: "32a56cab30ba679051a6d07ad4b7206f"
server_key_version: "encryptionkey@0"
I20260928 02:58:06.993885 18932 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance:
uuid: "2a6db1624773436eaceb9f696d8b74b3"
format_stamp: "Formatted at 2026-09-28 02:58:06 on dist-test-slave-s1kg"
server_key: "42f89902d112015308efcdb9a47e1100"
server_key_iv: "32a56cab30ba679051a6d07ad4b7206f"
server_key_version: "encryptionkey@0"
I20260928 02:58:06.999922 18932 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260928 02:58:07.004246 18947 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:07.005921 18932 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260928 02:58:07.006196 18932 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal
uuid: "2a6db1624773436eaceb9f696d8b74b3"
format_stamp: "Formatted at 2026-09-28 02:58:06 on dist-test-slave-s1kg"
server_key: "42f89902d112015308efcdb9a47e1100"
server_key_iv: "32a56cab30ba679051a6d07ad4b7206f"
server_key_version: "encryptionkey@0"
I20260928 02:58:07.006484 18932 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:07.027588 18932 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:07.032972 18932 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:07.033326 18932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:07.054505 18932 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.254:39307
I20260928 02:58:07.054530 18998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.254:39307 every 8 connection(s)
I20260928 02:58:07.056057 18932 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/data/info.pb
I20260928 02:58:07.060586 18999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:07.060817 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 18932
I20260928 02:58:07.061007 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-0/wal/instance
I20260928 02:58:07.061635 28567 external_mini_cluster.cc:1428] Setting key 68d2b328fb382b7922c5e7938e543b2a
I20260928 02:58:07.068536 18999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:07 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.229.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.253@KRBTEST.COM" created.
I20260928 02:58:07.092803 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.072367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38162 (local address 127.27.229.254:39307)
0928 02:58:07.074116 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:07.074128 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:07.074181 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:58:07.075949 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:07.075955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:07.076219 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:07.076432 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:07.077753 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.079040 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.080447 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.081018 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.081230 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.085240 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:07.085249 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:07.085263 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:07.085292 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:07.088133 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.088923 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.088931 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.088937 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.089090 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.089511 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.089518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.089522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.090052 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:07.090270 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:07.090778 (+ 508us) server_negotiation.cc:299] Negotiation successful
0928 02:58:07.091261 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":1534,"thread_start_us":1322,"threads_started":1}
I20260928 02:58:07.092823 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.072367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39307 (local address 127.0.0.1:38162)
0928 02:58:07.073282 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.073332 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:07.075528 (+ 2196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:07.076449 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:07.076455 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:07.077188 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:07.077529 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.077550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.079202 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.079209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.080300 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.080312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.081262 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.081269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.081435 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.082256 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:07.082280 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:07.084979 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:07.088296 (+ 3317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.088307 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.088320 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.088789 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.089208 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.089215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.089218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.089403 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.090197 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:07.090207 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:07.090412 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 02:58:07.090666 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 02:58:07.091391 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":177,"thread_start_us":411,"threads_started":1}
I20260928 02:58:07.096153 18999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:07.097474 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.096902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50858)
0928 02:58:07.097136 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.097287 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 02:58:07.097767 18949 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
I20260928 02:58:07.099617 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.099137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50866)
0928 02:58:07.099323 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.099419 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 02:58:07.099941 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40471: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
Entry for principal kudu/127.27.229.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab.
I20260928 02:58:07.122714 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40471
--webserver_interface=127.27.229.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:58:07.149331 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } attempt: 1
I20260928 02:58:07.150630 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.150081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50878)
0928 02:58:07.150308 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.150455 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260928 02:58:07.151791 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.151393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50888)
0928 02:58:07.151549 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.151637 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:58:07.152002 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40471: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
I20260928 02:58:07.220297 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } attempt: 2
I20260928 02:58:07.221725 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.221038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50892)
0928 02:58:07.221433 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.221542 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217}
I20260928 02:58:07.223013 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.222522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50906)
0928 02:58:07.222704 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.222774 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 02:58:07.223229 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40471: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
I20260928 02:58:07.314514 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } attempt: 3
I20260928 02:58:07.315795 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.315316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50920)
0928 02:58:07.315538 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.315640 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260928 02:58:07.316931 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.316555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50934)
0928 02:58:07.316714 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.316793 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:58:07.317149 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40471: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
I20260928 02:58:07.460438 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } attempt: 4
I20260928 02:58:07.461719 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.461126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50940)
0928 02:58:07.461415 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.461533 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 02:58:07.462740 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.462392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50946)
0928 02:58:07.462545 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.462614 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:58:07.462945 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.253:40471: Network error: Client connection negotiation failed: client connection to 127.27.229.253:40471: connect: Connection refused (error 111)
W20260928 02:58:07.469111 19006 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:07.469473 19006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:07.469591 19006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:07.469750 19006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 02:58:07.477286 19006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:07.477434 19006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:07.477540 19006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:07.477636 19006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:07.477732 19006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:07.488322 19006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.253.keytab
--principal=kudu/127.27.229.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.253:40471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.229.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:07.489775 19006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:07.491114 19006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:07.503378 19011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:07.503521 19012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:07.503405 19014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:07.504475 19006 server_base.cc:1034] running on GCE node
I20260928 02:58:07.505239 19006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:07.506731 19006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:07.508033 19006 hybrid_clock.cc:648] HybridClock initialized: now 1790564287507940 us; error 59 us; skew 500 ppm
I20260928 02:58:07.508353 19006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:07 dist-test-slave-s1kg krb5kdc[18919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:07.511909 19006 init.cc:377] Logged in from keytab as kudu/127.27.229.253@KRBTEST.COM (short username kudu)
I20260928 02:58:07.513481 19006 webserver.cc:460] Webserver started at http://127.27.229.253:39399/ using document root <none> and password file <none>
I20260928 02:58:07.514357 19006 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:07.514456 19006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:07.514798 19006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:07.517908 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data/instance:
uuid: "cb769cda0ef244b79383314972f26bed"
format_stamp: "Formatted at 2026-09-28 02:58:07 on dist-test-slave-s1kg"
server_key: "e8705c3093dae9d341c9b12ee6d8a0e4"
server_key_iv: "dbda8a023ca9a83c8f14b5cf020e42b8"
server_key_version: "encryptionkey@0"
I20260928 02:58:07.518721 19006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal/instance:
uuid: "cb769cda0ef244b79383314972f26bed"
format_stamp: "Formatted at 2026-09-28 02:58:07 on dist-test-slave-s1kg"
server_key: "e8705c3093dae9d341c9b12ee6d8a0e4"
server_key_iv: "dbda8a023ca9a83c8f14b5cf020e42b8"
server_key_version: "encryptionkey@0"
I20260928 02:58:07.524061 19006 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 02:58:07.527767 19021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:07.529147 19006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:07.529351 19006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal
uuid: "cb769cda0ef244b79383314972f26bed"
format_stamp: "Formatted at 2026-09-28 02:58:07 on dist-test-slave-s1kg"
server_key: "e8705c3093dae9d341c9b12ee6d8a0e4"
server_key_iv: "dbda8a023ca9a83c8f14b5cf020e42b8"
server_key_version: "encryptionkey@0"
I20260928 02:58:07.529589 19006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:07.552902 19006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:07.557322 19006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:07.557613 19006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:07.577071 19006 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.253:40471
I20260928 02:58:07.577097 19072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.253:40471 every 8 connection(s)
I20260928 02:58:07.578477 19006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/data/info.pb
I20260928 02:58:07.580410 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 19006
I20260928 02:58:07.580624 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-1/wal/instance
I20260928 02:58:07.581092 28567 external_mini_cluster.cc:1428] Setting key c25a761ab9f0c3f96be39b04ccf28ace
I20260928 02:58:07.582484 19073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:07.589177 19073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.229.252@KRBTEST.COM; defaulting to no policy
Sep 28 02:58:07 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.229.252@KRBTEST.COM" created.
I20260928 02:58:07.612128 19003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.592035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38176 (local address 127.27.229.254:39307)
0928 02:58:07.592216 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:07.592221 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:07.593678 (+ 1457us) server_negotiation.cc:407] Connection header received
0928 02:58:07.595737 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:07.595743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:07.595841 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:07.595956 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:07.597815 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.598592 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.600395 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.600968 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.601138 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.605306 (+ 4168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:07.605314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:07.605318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:07.605355 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:07.608183 (+ 2828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.609321 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.609326 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.609329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.609467 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.610006 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.610009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.610012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.610325 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:07.610451 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:07.611561 (+ 1110us) server_negotiation.cc:299] Negotiation successful
0928 02:58:07.611821 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 02:58:07.613432 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.592700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39307 (local address 127.0.0.1:38176)
0928 02:58:07.593502 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.593546 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:07.595445 (+ 1899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:07.596214 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:07.596223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:07.596882 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:07.597533 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.597552 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.598793 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.598797 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.600219 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.600233 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.601241 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.601248 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.601530 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.602309 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:07.602336 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:07.604955 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:07.608367 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.608382 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.608406 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.609144 (+ 738us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.609602 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.609610 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.609617 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.609870 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.610466 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:07.610477 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:07.611059 (+ 582us) client_negotiation.cc:769] Sending connection context
0928 02:58:07.611371 (+ 312us) client_negotiation.cc:240] Negotiation successful
0928 02:58:07.611954 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260928 02:58:07.615265 19073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:07 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:07.630576 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.616082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50960)
0928 02:58:07.616361 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.616380 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:07.616500 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:07.617566 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:07.617571 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:07.618020 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:07.618153 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.618162 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.619636 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.619641 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.620550 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.620562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.621398 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.621404 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.621633 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.622252 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:07.622270 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:07.624390 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:07.627848 (+ 3458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.627853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.627857 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.628318 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.628748 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.628752 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.628756 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.628884 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.629474 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:07.629479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:07.629615 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 02:58:07.629771 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 02:58:07.630064 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":43}
I20260928 02:58:07.630605 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.616443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50960 (local address 127.27.229.253:40471)
0928 02:58:07.617138 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:07.617144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:07.617170 (+ 26us) server_negotiation.cc:407] Connection header received
0928 02:58:07.617217 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:07.617223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:07.617408 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:07.617576 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:07.618340 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.619500 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.620698 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.621252 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.621393 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.624615 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:07.624627 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:07.624641 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:07.624673 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:07.627693 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.628476 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.628489 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.628491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.628602 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.628986 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.628994 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.628996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.629354 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:07.629552 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:07.629916 (+ 364us) server_negotiation.cc:299] Negotiation successful
0928 02:58:07.630187 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":544,"thread_start_us":393,"threads_started":1}
I20260928 02:58:07.633066 19073 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:07.634265 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.633721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56356)
0928 02:58:07.633900 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.634041 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 02:58:07.634521 19023 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.636232 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.635901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56368)
0928 02:58:07.636047 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.636122 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260928 02:58:07.636456 19073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
Entry for principal kudu/127.27.229.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab.
I20260928 02:58:07.642781 28567 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskhPicry/build/asan/bin/kudu
/tmp/dist-test-taskhPicry/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:36261
--webserver_interface=127.27.229.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.229.212:46761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 02:58:07.685797 19073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 1
I20260928 02:58:07.687100 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.686551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56384)
0928 02:58:07.686755 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.686938 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:58:07.688118 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.687750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56400)
0928 02:58:07.687921 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.688006 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 02:58:07.688365 19073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.756667 19073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 2
I20260928 02:58:07.757916 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.757442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56408)
0928 02:58:07.757663 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.757758 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260928 02:58:07.758989 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.758564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56418)
0928 02:58:07.758718 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.758792 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 02:58:07.759210 19073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.762199 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } attempt: 5
Sep 28 02:58:07 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:07.777370 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.762987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50962 (local address 127.27.229.253:40471)
0928 02:58:07.763163 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:07.763169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:07.763204 (+ 35us) server_negotiation.cc:407] Connection header received
0928 02:58:07.763364 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:07.763369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:07.763449 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:07.763562 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:07.764471 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.765586 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.767058 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.767732 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.767868 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.771464 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:07.771473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:07.771480 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:07.771534 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:07.774654 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.775441 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.775446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.775448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.775563 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:07.775956 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:07.775961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:07.775964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:07.776259 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:07.776419 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:07.776820 (+ 401us) server_negotiation.cc:299] Negotiation successful
0928 02:58:07.777044 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 02:58:07.777395 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.762943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50962)
0928 02:58:07.763111 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.763130 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:07.763246 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:07.763707 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:07.763711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:07.764113 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:07.764287 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.764297 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.765781 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.765785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.766787 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:07.766800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:07.767948 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:07.767952 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:07.768107 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:07.768820 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:07.768839 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:07.771197 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:07.774896 (+ 3699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.774902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.774906 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.775294 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.775681 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:07.775686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:07.775689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:07.775838 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:07.776384 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:07.776389 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:07.776547 (+ 158us) client_negotiation.cc:769] Sending connection context
0928 02:58:07.776688 (+ 141us) client_negotiation.cc:240] Negotiation successful
0928 02:58:07.777025 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 02:58:07.778626 18999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:07.779767 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.779361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56426)
0928 02:58:07.779542 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.779622 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 02:58:07.780700 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.780362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56440)
0928 02:58:07.780507 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.780596 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 02:58:07.780926 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.832223 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 1
I20260928 02:58:07.833734 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.833062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56456)
0928 02:58:07.833340 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.833488 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 02:58:07.835078 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.834618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56468)
0928 02:58:07.834806 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.834955 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 02:58:07.835381 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.850485 19073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 3
I20260928 02:58:07.851832 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.851219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56470)
0928 02:58:07.851486 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.851612 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 02:58:07.852999 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.852512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56482)
0928 02:58:07.852760 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.852855 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260928 02:58:07.853251 19073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.903630 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 2
I20260928 02:58:07.905009 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.904438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56494)
0928 02:58:07.904696 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.904848 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260928 02:58:07.906188 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.905779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56496)
0928 02:58:07.905969 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.906043 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260928 02:58:07.906409 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:07.996501 19073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 4
W20260928 02:58:07.997052 19080 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 02:58:07.997414 19080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 02:58:07.997524 19080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 02:58:07.997649 19080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 02:58:07.997896 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.997275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56504)
0928 02:58:07.997585 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.997695 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260928 02:58:07.998977 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:07.998569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56508)
0928 02:58:07.998721 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:07.998857 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 02:58:07.999233 19073 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
W20260928 02:58:08.005575 19080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 02:58:08.005716 19080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 02:58:08.005848 19080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 02:58:08.005935 19080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 02:58:08.006049 19080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 02:58:08.012661 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 3
I20260928 02:58:08.013821 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.013343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56510)
0928 02:58:08.013552 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.013658 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 02:58:08.014786 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.014456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56512)
0928 02:58:08.014604 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.014683 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 02:58:08.015030 18999 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.229.252:36261: Network error: Client connection negotiation failed: client connection to 127.27.229.252:36261: connect: Connection refused (error 111)
I20260928 02:58:08.017408 19080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.229.212:46761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.229.254:39307,127.27.229.253:40471,127.27.229.252:36261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/krb5kdc/kudu_127.27.229.252.keytab
--principal=kudu/127.27.229.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.229.252:36261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.229.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 28 Sep 2026 02:02:13 UTC on ccycloud.quasar-pwdsdm.root.comops.site
build id 4676526
ASAN enabled
I20260928 02:58:08.018967 19080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 02:58:08.020421 19080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 02:58:08.033308 19088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:08.033371 19085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 02:58:08.033699 19086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 02:58:08.034931 19080 server_base.cc:1034] running on GCE node
I20260928 02:58:08.035809 19080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 02:58:08.037406 19080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 02:58:08.038702 19080 hybrid_clock.cc:648] HybridClock initialized: now 1790564288038648 us; error 85 us; skew 500 ppm
I20260928 02:58:08.039144 19080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 02:58:08.042570 19080 init.cc:377] Logged in from keytab as kudu/127.27.229.252@KRBTEST.COM (short username kudu)
I20260928 02:58:08.044199 19080 webserver.cc:460] Webserver started at http://127.27.229.252:35277/ using document root <none> and password file <none>
I20260928 02:58:08.044957 19080 fs_manager.cc:359] Metadata directory not provided
I20260928 02:58:08.045094 19080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 02:58:08.045413 19080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 02:58:08.048620 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data/instance:
uuid: "d1b1e2aefe0c4389ad63f13379dc72db"
format_stamp: "Formatted at 2026-09-28 02:58:08 on dist-test-slave-s1kg"
server_key: "e55a54ff6d814db0cae25eed922fbfd7"
server_key_iv: "3ba1e1f73ec1c3b52b09fe4b8f854b33"
server_key_version: "encryptionkey@0"
I20260928 02:58:08.049451 19080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal/instance:
uuid: "d1b1e2aefe0c4389ad63f13379dc72db"
format_stamp: "Formatted at 2026-09-28 02:58:08 on dist-test-slave-s1kg"
server_key: "e55a54ff6d814db0cae25eed922fbfd7"
server_key_iv: "3ba1e1f73ec1c3b52b09fe4b8f854b33"
server_key_version: "encryptionkey@0"
I20260928 02:58:08.055188 19080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 02:58:08.059741 19095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 02:58:08.061395 19080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 02:58:08.061596 19080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data,/tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal
uuid: "d1b1e2aefe0c4389ad63f13379dc72db"
format_stamp: "Formatted at 2026-09-28 02:58:08 on dist-test-slave-s1kg"
server_key: "e55a54ff6d814db0cae25eed922fbfd7"
server_key_iv: "3ba1e1f73ec1c3b52b09fe4b8f854b33"
server_key_version: "encryptionkey@0"
I20260928 02:58:08.061869 19080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 02:58:08.115116 19080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 02:58:08.119789 19080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 02:58:08.120096 19080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 02:58:08.139844 19080 rpc_server.cc:307] RPC server started. Bound to: 127.27.229.252:36261
I20260928 02:58:08.139868 19146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.229.252:36261 every 8 connection(s)
I20260928 02:58:08.141208 19080 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/data/info.pb
I20260928 02:58:08.145006 19147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 02:58:08.150998 28567 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskhPicry/build/asan/bin/kudu as pid 19080
I20260928 02:58:08.151194 28567 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskhPicry/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790563915649057-28567-0/minicluster-data/master-2/wal/instance
I20260928 02:58:08.151714 28567 external_mini_cluster.cc:1428] Setting key cf707ed547ab679ae0c874c7b80595fd
I20260928 02:58:08.152056 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:08.172951 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.155075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.27.229.254:39307)
0928 02:58:08.156590 (+ 1515us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.156596 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.156616 (+ 20us) server_negotiation.cc:407] Connection header received
0928 02:58:08.157312 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.157316 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.157383 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.157506 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.158917 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.159760 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.161448 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.162023 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.162117 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.166219 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.166226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.166231 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.166274 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.169719 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.170476 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.170482 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.170486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.170627 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.171136 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.171142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.171145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.171434 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.171592 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.172283 (+ 691us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.172523 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1393,"thread_start_us":1283,"threads_started":1}
I20260928 02:58:08.173979 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.155038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.254:39307 (local address 127.0.0.1:38188)
0928 02:58:08.155818 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.155865 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:08.157020 (+ 1155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:08.157575 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:08.157583 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:08.158356 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:08.158715 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.158732 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.159926 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.159932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.161298 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.161310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.162285 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.162293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.162559 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.163430 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:08.163452 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:08.165948 (+ 2496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:08.169885 (+ 3937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.169897 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.169909 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.170350 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.170744 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.170749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.170752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.171018 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.171562 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:08.171570 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:08.171936 (+ 366us) client_negotiation.cc:769] Sending connection context
0928 02:58:08.172179 (+ 243us) client_negotiation.cc:240] Negotiation successful
0928 02:58:08.172700 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1,"thread_start_us":349,"threads_started":1}
I20260928 02:58:08.173997 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.153707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.27.229.252:36261)
0928 02:58:08.154479 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.154490 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.154543 (+ 53us) server_negotiation.cc:407] Connection header received
0928 02:58:08.157020 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.157049 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.157363 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.157571 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.159042 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.160428 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.161861 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.162605 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.162790 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.167313 (+ 4523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.167321 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.167335 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.167365 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.170135 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.170882 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.170894 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.170896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.171028 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.171413 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.171418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.171422 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.171974 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.172291 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.172614 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.172880 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"spinlock_wait_cycles":16640,"thread_start_us":365,"threads_started":1}
I20260928 02:58:08.176061 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.254@KRBTEST.COM
I20260928 02:58:08.192276 18999 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 4
I20260928 02:58:08.196060 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.177203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.253:40471 (local address 127.0.0.1:50966)
0928 02:58:08.177611 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.177664 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:08.177929 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:08.178347 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:08.178352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:08.178812 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:08.179015 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.179024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.180379 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.180386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.181623 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.181640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.182646 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.182651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.182950 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.187049 (+ 4099us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:08.187070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:08.189280 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:08.193075 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.193084 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.193088 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.193643 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.194079 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.194085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.194089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.194270 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.195038 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:08.195042 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:08.195211 (+ 169us) client_negotiation.cc:769] Sending connection context
0928 02:58:08.195373 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 02:58:08.195680 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312}
I20260928 02:58:08.196079 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.177193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.27.229.253:40471)
0928 02:58:08.177416 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.177486 (+ 70us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.177796 (+ 310us) server_negotiation.cc:407] Connection header received
0928 02:58:08.178089 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.178097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.178192 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.178284 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.179171 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.180223 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.181793 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.182466 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.182613 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.189596 (+ 6983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.189606 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.189613 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.189661 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.192800 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.193789 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.193794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.193798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.193982 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.194380 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.194386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.194390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.194782 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.194970 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.195465 (+ 495us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.195681 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 02:58:08.197399 19147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } has no permanent_uuid. Determining permanent_uuid...
I20260928 02:58:08.198431 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.178022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.27.229.254:39307)
0928 02:58:08.178207 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.178213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.178236 (+ 23us) server_negotiation.cc:407] Connection header received
0928 02:58:08.178312 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.178317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.178404 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.178510 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.180067 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.181099 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.182755 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.183405 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.183557 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.190582 (+ 7025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.190588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.190591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.190626 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.194169 (+ 3543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.194914 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.194919 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.194921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.195074 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.195504 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.195510 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.195514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.195854 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.197691 (+ 1837us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.197835 (+ 144us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.198075 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.254@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564288, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.252@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:08.209964 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.193201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56524 (local address 127.27.229.252:36261)
0928 02:58:08.193374 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.193381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.193423 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:58:08.193625 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.193631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.193702 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.193877 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.194864 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.195894 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.197751 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.198457 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.199766 (+ 1309us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.203086 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.203099 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.203103 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.203133 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.206518 (+ 3385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.207593 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.207598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.207602 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.207775 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.208304 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.208309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.208312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.208733 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.208911 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.209396 (+ 485us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.209614 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 02:58:08.210005 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.193046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56524)
0928 02:58:08.193352 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.193373 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:08.193493 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:08.193936 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:08.193942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:08.194455 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:08.194667 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.194681 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.196037 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.196042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.197543 (+ 1501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.197560 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.199128 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.199136 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.199373 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.200058 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:08.200087 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:08.202886 (+ 2799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:08.206725 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.206734 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.206738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.207416 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.207939 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.207944 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.207949 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.208151 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.208897 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:08.208907 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:08.209100 (+ 193us) client_negotiation.cc:769] Sending connection context
0928 02:58:08.209296 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 02:58:08.209605 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.229.253@KRBTEST.COM
I20260928 02:58:08.215869 18999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: Bootstrap starting.
I20260928 02:58:08.216524 19151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.198175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56526)
0928 02:58:08.198468 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.198769 (+ 301us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:08.198925 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:08.199767 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:08.199772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:08.200263 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:08.200454 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.200465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.202124 (+ 1659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.202129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.203641 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.203658 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.204508 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.204513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.204796 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.205423 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:08.205446 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:08.208086 (+ 2640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:08.213012 (+ 4926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.213017 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.213021 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.213529 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.213965 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.213970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.213973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.214123 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.215407 (+ 1284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:08.215412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:08.215583 (+ 171us) client_negotiation.cc:769] Sending connection context
0928 02:58:08.215741 (+ 158us) client_negotiation.cc:240] Negotiation successful
0928 02:58:08.216077 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1}
I20260928 02:58:08.216742 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.198215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56526 (local address 127.27.229.252:36261)
0928 02:58:08.199480 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.199488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.199512 (+ 24us) server_negotiation.cc:407] Connection header received
0928 02:58:08.199556 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.199561 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.199628 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.200585 (+ 957us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.200832 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.201962 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.203763 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.204336 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.204497 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.208307 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.208313 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.208316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.208346 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.212874 (+ 4528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.213677 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.213683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.213686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.213842 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.214271 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.214279 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.214283 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.215265 (+ 982us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.215426 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.215907 (+ 481us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.216242 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":1132,"spinlock_wait_cycles":1280,"thread_start_us":983,"threads_started":1}
I20260928 02:58:08.219467 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.200347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50968 (local address 127.27.229.253:40471)
0928 02:58:08.200519 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.200526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.200568 (+ 42us) server_negotiation.cc:407] Connection header received
0928 02:58:08.200778 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.200785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.200873 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.200984 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.202405 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.203528 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.205421 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.206090 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.206219 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.212431 (+ 6212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.212440 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.212449 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.212494 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.215705 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.216936 (+ 1231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.216942 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.216945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.217109 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.217623 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.217628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.217630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.217992 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.218258 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.218846 (+ 588us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.219096 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 02:58:08.219663 18999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:08.220819 18999 log.cc:824] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:08.221907 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: Bootstrap starting.
I20260928 02:58:08.223793 18999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: No bootstrap required, opened a new log
I20260928 02:58:08.225495 19147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:08.226907 19147 log.cc:824] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:08.228247 28567 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 02:58:08.232147 18974 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2a6db1624773436eaceb9f696d8b74b3"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38204
I20260928 02:58:08.232901 18999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.233207 18999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a6db1624773436eaceb9f696d8b74b3, State: Initialized, Role: FOLLOWER
I20260928 02:58:08.233656 18999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.234535 19147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: No bootstrap required, opened a new log
I20260928 02:58:08.234705 19156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.235141 19156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.235981 18999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:08.236032 18974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 02:58:08.236213 18974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:08.241030 18974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.243276 19147 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.243652 19147 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b1e2aefe0c4389ad63f13379dc72db, State: Initialized, Role: FOLLOWER
I20260928 02:58:08.244191 18974 leader_election.cc:288] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [CANDIDATE]: Term 1 election: Requested vote from peers cb769cda0ef244b79383314972f26bed (127.27.229.253:40471), d1b1e2aefe0c4389ad63f13379dc72db (127.27.229.252:36261)
I20260928 02:58:08.244223 19147 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.251894 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.252409 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.253849 19147 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:08.253945 19122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a6db1624773436eaceb9f696d8b74b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d1b1e2aefe0c4389ad63f13379dc72db"
I20260928 02:58:08.254606 19122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:08.260211 19122 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a6db1624773436eaceb9f696d8b74b3 in term 1.
I20260928 02:58:08.261456 18949 leader_election.cc:302] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a6db1624773436eaceb9f696d8b74b3, d1b1e2aefe0c4389ad63f13379dc72db; no voters:
I20260928 02:58:08.261650 19173 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:58:08.262032 19156 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 02:58:08.271538 19156 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [term 1 LEADER]: Becoming Leader. State: Replica: 2a6db1624773436eaceb9f696d8b74b3, State: Running, Role: LEADER
I20260928 02:58:08.272513 19156 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.285863 19077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.261297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.27.229.253:40471)
0928 02:58:08.261503 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.261509 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.263549 (+ 2040us) server_negotiation.cc:407] Connection header received
0928 02:58:08.263974 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.263983 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.264072 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.264152 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.267503 (+ 3351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.269102 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.271969 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.272916 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.273060 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.277802 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.277812 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.277817 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.277871 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.281915 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.283281 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.283288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.283291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.283468 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.283942 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.283948 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.283952 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.284349 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.284644 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.285226 (+ 582us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.285490 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260928 02:58:08.286504 18953 tablet.cc:2369] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:08.290311 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6db1624773436eaceb9f696d8b74b3. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.291098 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:08.298471 19073 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } attempt: 5
I20260928 02:58:08.314579 19153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.263578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56530 (local address 127.27.229.252:36261)
0928 02:58:08.263957 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.263964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.275237 (+ 11273us) server_negotiation.cc:407] Connection header received
0928 02:58:08.278631 (+ 3394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.278638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.278732 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.278853 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.284653 (+ 5800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.285789 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.293344 (+ 7555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.294440 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.294600 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.299760 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.299771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.299779 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.299839 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.304496 (+ 4657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.311194 (+ 6698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.311205 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.311211 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.311479 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.312066 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.312074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.312080 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.312672 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.313003 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.313722 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.314071 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260928 02:58:08.316120 19182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 02:58:08 dist-test-slave-s1kg krb5kdc[18919](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790564287, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.229.253@KRBTEST.COM for kudu/127.27.229.252@KRBTEST.COM
I20260928 02:58:08.292035 18932 master_runner.cc:186] Error getting master registration for 127.27.229.253:40471: OK, instance_id { permanent_uuid: "cb769cda0ef244b79383314972f26bed" instance_seqno: 1790564287571602 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:58:08.320749 18932 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:58:08.322906 19152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.271102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38214 (local address 127.27.229.254:39307)
0928 02:58:08.271326 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.271333 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.276018 (+ 4685us) server_negotiation.cc:407] Connection header received
0928 02:58:08.276788 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.276796 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.276895 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.276970 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.281499 (+ 4529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.282695 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.288761 (+ 6066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.289677 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.289840 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.299066 (+ 9226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.299078 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.299087 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.299157 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.318385 (+ 19228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.319423 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.319428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.319432 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.319632 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.320132 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.320138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.320143 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.320604 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.322037 (+ 1433us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.322192 (+ 155us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.322477 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:58:08.326215 19076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.301021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.229.252:36261 (local address 127.0.0.1:56532)
0928 02:58:08.301379 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 02:58:08.301405 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 02:58:08.301564 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 02:58:08.302604 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 02:58:08.302613 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 02:58:08.303352 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0928 02:58:08.303659 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.303678 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.305818 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.305827 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.307913 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 02:58:08.307937 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.311557 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.311568 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 02:58:08.311956 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.315202 (+ 3246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 02:58:08.315241 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 02:58:08.318059 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 02:58:08.322577 (+ 4518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.322583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.322587 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.323554 (+ 967us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.324048 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 02:58:08.324053 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 02:58:08.324056 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 02:58:08.324269 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 02:58:08.325115 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 02:58:08.325120 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 02:58:08.325283 (+ 163us) client_negotiation.cc:769] Sending connection context
0928 02:58:08.325425 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 02:58:08.325772 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188}
I20260928 02:58:08.327399 19150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:08.301173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56532 (local address 127.27.229.252:36261)
0928 02:58:08.301395 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.301401 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.301698 (+ 297us) server_negotiation.cc:407] Connection header received
0928 02:58:08.301781 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.301790 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.301897 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.302011 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.304059 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.305571 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.308190 (+ 2619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:08.309210 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:08.309381 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:08.318277 (+ 8896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 02:58:08.318286 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 02:58:08.318293 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 02:58:08.318332 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 02:58:08.322388 (+ 4056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.323726 (+ 1338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.323734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.323737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.323895 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 02:58:08.324411 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 02:58:08.324417 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 02:58:08.324420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 02:58:08.324861 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 02:58:08.325050 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:08.326710 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0928 02:58:08.326997 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 02:58:08.328341 19182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 02:58:08.333426 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: Bootstrap starting.
I20260928 02:58:08.342399 19073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: Neither blocks nor log segments found. Creating new log.
I20260928 02:58:08.344082 19073 log.cc:824] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: Log is configured to *not* fsync() on all Append() calls
I20260928 02:58:08.358662 19073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: No bootstrap required, opened a new log
W20260928 02:58:08.364472 19192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 02:58:08.364696 19192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 02:58:08.372455 19101 tablet.cc:2369] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 02:58:08.380967 19122 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [term 1 FOLLOWER]: Refusing update from remote peer 2a6db1624773436eaceb9f696d8b74b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:08.382042 19175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:08.385728 19073 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.386179 19073 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb769cda0ef244b79383314972f26bed, State: Initialized, Role: FOLLOWER
I20260928 02:58:08.386752 19073 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } }
I20260928 02:58:08.388280 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.389508 19073 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: configured and running, proceeding with master startup.
I20260928 02:58:08.389935 19048 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a6db1624773436eaceb9f696d8b74b3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "cb769cda0ef244b79383314972f26bed"
I20260928 02:58:08.390455 19048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [term 0 FOLLOWER]: Advancing to term 1
I20260928 02:58:08.366128 19080 master_runner.cc:186] Error getting master registration for 127.27.229.253:40471: OK, instance_id { permanent_uuid: "cb769cda0ef244b79383314972f26bed" instance_seqno: 1790564287571602 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 02:58:08.391299 19080 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 02:58:08.398217 19048 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a6db1624773436eaceb9f696d8b74b3 in term 1.
I20260928 02:58:08.398983 19047 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [term 1 FOLLOWER]: Refusing update from remote peer 2a6db1624773436eaceb9f696d8b74b3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 02:58:08.401081 19175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 02:58:08.405704 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.408013 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6db1624773436eaceb9f696d8b74b3. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.408430 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.411944 19187 mvcc.cc:205] Tried to move back new op lower bound from 7334151325185380352 to 7334151324771438592. Current Snapshot: MvccSnapshot[applied={T|T < 7334151325185380352}]
I20260928 02:58:08.419750 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.420219 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.429402 19156 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.429804 19156 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:08.430611 19182 catalog_manager.cc:1347] Generated new cluster ID: a57f031bebcd4eb2bd2d987d475c97bc
I20260928 02:58:08.430728 19182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 02:58:08.445573 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a6db1624773436eaceb9f696d8b74b3. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.445951 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.461000 19175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.461331 19175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3 [sys.catalog]: This master's current role is: LEADER
I20260928 02:58:08.463737 19204 mvcc.cc:205] Tried to move back new op lower bound from 7334151325185380352 to 7334151324771438592. Current Snapshot: MvccSnapshot[applied={T|T < 7334151325185380352}]
I20260928 02:58:08.467514 19194 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a6db1624773436eaceb9f696d8b74b3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a6db1624773436eaceb9f696d8b74b3" member_type: VOTER last_known_addr { host: "127.27.229.254" port: 39307 } } peers { permanent_uuid: "cb769cda0ef244b79383314972f26bed" member_type: VOTER last_known_addr { host: "127.27.229.253" port: 40471 } } peers { permanent_uuid: "d1b1e2aefe0c4389ad63f13379dc72db" member_type: VOTER last_known_addr { host: "127.27.229.252" port: 36261 } } }
I20260928 02:58:08.467826 19194 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed [sys.catalog]: This master's current role is: FOLLOWER
I20260928 02:58:08.496194 19182 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 02:58:08.498206 19182 catalog_manager.cc:1504] Loading token signing keys...
I20260928 02:58:08.499390 19214 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 02:58:08.516994 19182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2a6db1624773436eaceb9f696d8b74b3: Generated new TSK 0
I20260928 02:58:08.518170 19182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 02:58:08.619288 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.609383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39682 (local address 127.27.229.253:40471)
0928 02:58:08.609605 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.609613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.616473 (+ 6860us) server_negotiation.cc:407] Connection header received
0928 02:58:08.617134 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.617140 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.617243 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.617321 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.619004 (+ 1683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39682: BlockingRecv error: recv got EOF from 127.0.0.1:39682 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260928 02:58:08.621732 19152 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.609036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38418 (local address 127.27.229.254:39307)
0928 02:58:08.609282 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.609289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.619983 (+ 10694us) server_negotiation.cc:407] Connection header received
0928 02:58:08.620249 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.620255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.620349 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.620435 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.621456 (+ 1021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38418: BlockingRecv error: recv got EOF from 127.0.0.1:38418 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:58:08.623207 19150 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.620572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.27.229.252:36261)
0928 02:58:08.620735 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.620741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.621281 (+ 540us) server_negotiation.cc:407] Connection header received
0928 02:58:08.621558 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.621564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.621649 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.621737 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.622948 (+ 1211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39272: BlockingRecv error: recv got EOF from 127.0.0.1:39272 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 02:58:08.630081 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.627862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.27.229.253:40471)
0928 02:58:08.628041 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.628048 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.628562 (+ 514us) server_negotiation.cc:407] Connection header received
0928 02:58:08.628786 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.628793 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.628922 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.629037 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.629804 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39692: BlockingRecv error: recv got EOF from 127.0.0.1:39692 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 02:58:08.643517 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.641158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.27.229.253:40471)
0928 02:58:08.641348 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.641354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.641908 (+ 554us) server_negotiation.cc:407] Connection header received
0928 02:58:08.642092 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.642098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.642203 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.642317 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.643268 (+ 951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39706: BlockingRecv error: recv got EOF from 127.0.0.1:39706 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:58:08.652741 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.650813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39718 (local address 127.27.229.253:40471)
0928 02:58:08.651037 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.651043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.651119 (+ 76us) server_negotiation.cc:407] Connection header received
0928 02:58:08.651423 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.651431 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.651539 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.651651 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.652477 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39718: BlockingRecv error: recv got EOF from 127.0.0.1:39718 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
W20260928 02:58:08.667502 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.665063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.27.229.253:40471)
0928 02:58:08.665286 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.665292 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.665821 (+ 529us) server_negotiation.cc:407] Connection header received
0928 02:58:08.666108 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.666114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.666217 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.666327 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.667252 (+ 925us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39730: BlockingRecv error: recv got EOF from 127.0.0.1:39730 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260928 02:58:08.687409 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.685053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39742 (local address 127.27.229.253:40471)
0928 02:58:08.685244 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.685251 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.685825 (+ 574us) server_negotiation.cc:407] Connection header received
0928 02:58:08.686025 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.686034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.686135 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.686252 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.687159 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39742: BlockingRecv error: recv got EOF from 127.0.0.1:39742 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 02:58:08.709695 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.707279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39748 (local address 127.27.229.253:40471)
0928 02:58:08.707482 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.707489 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.708072 (+ 583us) server_negotiation.cc:407] Connection header received
0928 02:58:08.708309 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.708315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.708437 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.708554 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.709424 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39748: BlockingRecv error: recv got EOF from 127.0.0.1:39748 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260928 02:58:08.748498 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.746033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39764 (local address 127.27.229.253:40471)
0928 02:58:08.746235 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.746241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.746769 (+ 528us) server_negotiation.cc:407] Connection header received
0928 02:58:08.746997 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.747003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.747105 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.747231 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.748193 (+ 962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39764: BlockingRecv error: recv got EOF from 127.0.0.1:39764 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260928 02:58:08.826010 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.823590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39774 (local address 127.27.229.253:40471)
0928 02:58:08.823839 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.823846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.824431 (+ 585us) server_negotiation.cc:407] Connection header received
0928 02:58:08.824656 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.824662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.824778 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.824901 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.825776 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39774: BlockingRecv error: recv got EOF from 127.0.0.1:39774 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 02:58:08.960104 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:08.958182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.27.229.253:40471)
0928 02:58:08.958450 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:08.958459 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:08.958641 (+ 182us) server_negotiation.cc:407] Connection header received
0928 02:58:08.958944 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:08.958949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:08.959022 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:08.959130 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:08.959887 (+ 757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39782: BlockingRecv error: recv got EOF from 127.0.0.1:39782 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
W20260928 02:58:09.221720 19077 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 02:58:09.219786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.27.229.253:40471)
0928 02:58:09.220031 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.220037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.220190 (+ 153us) server_negotiation.cc:407] Connection header received
0928 02:58:09.220456 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.220466 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.220540 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.220655 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 02:58:09.221461 (+ 806us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39788: BlockingRecv error: recv got EOF from 127.0.0.1:39788 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260928 02:58:09.367156 19192 catalog_manager.cc:1259] Loaded cluster ID: a57f031bebcd4eb2bd2d987d475c97bc
I20260928 02:58:09.367285 19192 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: loading cluster ID for follower catalog manager: success
I20260928 02:58:09.370086 19192 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: acquiring CA information for follower catalog manager: success
I20260928 02:58:09.372356 19192 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d1b1e2aefe0c4389ad63f13379dc72db: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:58:09.504935 19214 catalog_manager.cc:1259] Loaded cluster ID: a57f031bebcd4eb2bd2d987d475c97bc
I20260928 02:58:09.505095 19214 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: loading cluster ID for follower catalog manager: success
I20260928 02:58:09.509130 19214 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: acquiring CA information for follower catalog manager: success
I20260928 02:58:09.511948 19214 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cb769cda0ef244b79383314972f26bed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 02:58:09.745983 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:09.739339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39792 (local address 127.27.229.253:40471)
0928 02:58:09.740121 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.740126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.740142 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:58:09.740206 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.740210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.740352 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.740496 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:09.740972 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.741925 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.743437 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.743975 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.744115 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:09.744641 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:09.745316 (+ 675us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:09.745490 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:09.745565 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 02:58:09.745663 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":447,"threads_started":1}
I20260928 02:58:09.760378 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:09.752078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39276 (local address 127.27.229.252:36261)
0928 02:58:09.754647 (+ 2569us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.754652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.754668 (+ 16us) server_negotiation.cc:407] Connection header received
0928 02:58:09.754760 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.754764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.755029 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.755201 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:09.755718 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.756433 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.757721 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.758127 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.758268 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:09.758889 (+ 621us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:09.759691 (+ 802us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:09.759833 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:09.759976 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 02:58:09.760051 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2464,"thread_start_us":2381,"threads_started":1}
I20260928 02:58:09.776636 19279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:09.768268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39808 (local address 127.27.229.253:40471)
0928 02:58:09.768453 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.768462 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.769435 (+ 973us) server_negotiation.cc:407] Connection header received
0928 02:58:09.769534 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.769539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.769649 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.769748 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:09.770395 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.771587 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.773816 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.774548 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.774695 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:09.775438 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:09.775583 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:09.775681 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:09.776172 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 02:58:09.776268 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 02:58:09.781551 19285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:09.769344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39288 (local address 127.27.229.252:36261)
0928 02:58:09.774679 (+ 5335us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.774684 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.774701 (+ 17us) server_negotiation.cc:407] Connection header received
0928 02:58:09.774762 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.774766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.774939 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.775046 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:09.776013 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.777279 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.779277 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.779950 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.780086 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:09.780755 (+ 669us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:09.780880 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:09.780988 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:09.781142 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 02:58:09.781229 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5190}
I20260928 02:58:09.783528 19291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 02:58:09.767899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38424 (local address 127.27.229.254:39307)
0928 02:58:09.769025 (+ 1126us) server_negotiation.cc:206] Beginning negotiation
0928 02:58:09.769032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 02:58:09.775510 (+ 6478us) server_negotiation.cc:407] Connection header received
0928 02:58:09.775874 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 02:58:09.775881 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 02:58:09.776131 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 02:58:09.776308 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 02:58:09.776973 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.778053 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.780747 (+ 2694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 02:58:09.781435 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 02:58:09.781564 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 02:58:09.782314 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 02:58:09.782790 (+ 476us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 02:58:09.782945 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 02:58:09.783142 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 02:58:09.783233 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":991,"thread_start_us":886,"threads_started":1}
I20260928 02:58:09.792281 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 18932
I20260928 02:58:09.793474 18994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:09.927170 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 18932
W20260928 02:58:09.948496 19098 connection.cc:535] server connection from 127.0.0.1:56524 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 02:58:09.948977 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 19006
I20260928 02:58:09.950073 19068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:10.059011 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 19006
I20260928 02:58:10.079820 28567 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskhPicry/build/asan/bin/kudu pid 19080
I20260928 02:58:10.081060 19142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260928 02:58:10.190133 28567 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskhPicry/build/asan/bin/kudu with pid 19080
2026-09-28T02:58:10Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4996 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9881 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (374507 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST